The Value of Personalization in Large Language Model Responses
Questrom School of Business and Faculty of Computing & Data Sciences, Boston University. September 2026.
Read the paper (PDF)Abstract
We study heterogeneous preferences for LLM responses and how personalized routing across models increases end-user utility. To do so, we estimate a random-coefficients demand model on Arena preference data and simulate the effects of a variety of model routers. A router which uses just the prompt raises predicted mean per-user utility over the best single model, attaining a 53.8% model-implied head-to-head win rate. Allowing the router to personalize to each user’s own taste vector, rather than serving a model catered to the average user’s pick, raises the win rate to 71.0%. The allocation of responses includes all models we consider, even though some have much higher benchmark scores than others. We then investigate the feasibility of learning individual preferences using repeat interactions. After forty feedback rounds, we can close 72% of the gap to the individual taste-based router. We also consider the efficacy of verbalizing estimated tastes. This approach yields a 70.4% predicted preference for the intended user versus a non-verbalized prompt. Lastly, we evaluate routers that choose a model before generation, and find these to be less effective. Even after forty past choices, this ex-ante router recovers just 35% of the ex-post gain.