Shifting Standards or Changing Preferences? Unraveling Review Polarization via LLMs

Status: Revise and resubmit at Journal of Marketing Research
Coauthors: Limin Fang (UBC), Baohong Sun (CKGSB)

Abstract

This paper investigates the drivers of online review polarization, characterized by a growing prevalence of extreme ratings (1-star and 5-star). Leveraging nearly seven million Yelp reviews across 11 metropolitan markets and using Large Language Models (LLMs) to analyze review content, we separate genuine changes in consumer experiences (content effect) from shifts in numerical rating standards (scale effect). We find that rising 5-star reviews primarily reflect scale inflation (65%), whereas increased 1-star ratings largely capture declining consumer experiences (80%). Significant contributors to scale shifts include temporal trends, explaining about half of the increase in extreme ratings, and reviewer heterogeneity. The scale effect reduces review informativeness by making numerical ratings less consistent with review content, disproportionately benefiting newer businesses and distorting market competition. Our findings suggest that review platforms should consider user-experience-weighted ratings or content-based AI-generated scores to enhance informativeness, consistency, and credibility.