Yelp may not be illegal, but the combination of the Yelp review filter, Yelp’s policy against asking for reviews, and the fact that most satisfied consumers don’t voluntarily write reviews, stacks the deck against small businesses and professionals…
Review site, Yelp, carefully positions itself as the purveyor and arbiter of “recommended reviews”. Do Yelp business practices and the review recommendation algorithm (aka the “review filter”) introduce inaccuracy and bias to the overall ratings for a business?
Spoiler alert: Yes, it does.
Many of the arguments about Yelp bias are legal ones (i.e. business owners seek access to anonymous reviewer identities, and accusations of review manipulation by Yelp sales). However, we believe that there is a fundamental mathematical bias to Yelp reviews that is often overlooked.
There’s no shortage of controversy surrounding Yelp reviews – many business owners feel that Yelp business practices can be unfair or even extortionary. In fact, Yelp receives six Federal subpoenas per month [1] on average from business owners, and Yelp will be the subject of an upcoming documentary, “Billion Dollar Bully“.
In a 2014 court case (Levitt vs Yelp [2]), it was ruled that Yelp is effectively free to manipulate reviews.
So with the above in mind, we ask the question, “Are Yelp reviews biased?”.
Some of the questions we’ll answer along the way…
- Do 1% of your customers control 100% of your online reputation?
- Sample size: how many reviews do you need?
- How do customer expectations bias Yelp reviews?
- Does the Yelp filter introduce bias?
- Does asking for reviews introduce bias?
- To ask or not to ask for reviews?
Full Disclosure: Real Word of Mouth, the author of this article, is a Yelp alternative, and provides advantages to business owners that are not available with Yelp. Some of those advantages are the basis for this article.
Do 1% of your customers control 100% of your online reputation?
Typically, less than 1% of your customers will voluntarily write an unsolicited review of your business. Do reviews from less than 1% of your customers fairly represent your business? Of course not – it doesn’t take a degree in statistics to intuitively understand that.
Many perfectly good businesses have been harmed by a few vociferous customers, and some have even been forced into closing their doors in spite of the fact that as much as 99% of their customers might be perfectly satisfied.
So how does it come to be that so few customers wield so much power over a business’ success or failure? The answer lies in statistics and something called insufficient “sample size”. The sample size (i.e. number of recommended reviews) in Yelp is influenced (biased) by:
- customer expectations
- filtered reviews
- no solicited reviews
Sample size: how many reviews do you need?
You probably need more reviews than you think.
The answer to this question is not intuitive because in statistics, the shapes of probability curves are anything but linear. A class in statistics would be helpful to understand this, but we’ll try to demonstrate this without diving into the math and underlying assumptions.
First, some definitions. In statistics, all the customers for a business would be referred to as the “population”. The group of customers that you want to measure (in this case, customers that write a review) would be referred to as the “sample”. And the number of customers in the sample is referred to as the “sample size”.
Now comes the non-intuitive part: a small business with fewer customers needs reviews from a LARGER PERCENTAGE of their customers than a large business with many customers. This really puts businesses with fewer customers at a statistical disadvantage.
Let’s demonstrate this using a handy online statistics calculator. The following calculations are not meant to calculate a precise number of reviews that you need. It’s really only meant to demonstrate how statistics favors large populations – in this case, larger businesses with more customers. See the Addendum at the end for more caveats regarding the sample size calculation.
You can see from the calculator that a business with 100 customers needs reviews from at least 49 customers to have a business review rating with some accuracy (+/-10%) and confidence (95%). That’s almost half of the business’ customers. Again, this calculation of 49 customers is a gross approximation at best – we’re simply trying to demonstrate the statistical disadvantage that small businesses have when it comes to getting an accurate review rating.
If we repeat the same calculation for a business with 1000 customers, you see that the business needs reviews from 88 customers for the same accuracy and confidence. That’s only 8.8% of 1000 customers, compared to 49% of 100 customers. Maybe a large business can get reviews from 8.8% of their customers without doing anything, but a small business needs an active review strategy in order to get reviews from 49% of their customers.
If you want more accuracy or higher confidence, even more reviews are needed (i.e. larger sample size). We include columns for higher accuracies in the table above to demonstrate this.
Unlike a restaurant or retailer with 1000 customers, a business with 100 customers like a contractor, architect, attorney, real estate agent, consultant or financial advisor, is much less likely to receive enough unsolicited reviews to reach an accurate review rating.
If the business could ask for reviews, it might be possible to get enough reviews to have an accurate business rating. But if the business owner can’t ask for reviews and has to rely entirely on unsolicited reviews, they are in an almost impossible position because of customer expectations.
Sample size: caveat
Again, the goal in the above calculations was to demonstrate how statistics of online reviews favors large populations.
Therefore, another take-away is that you need more than a handful or two of reviews to accurately represent your business.
How do customer expectations bias Yelp reviews?
Customers are more likely to write a review when their expectations are not met, or when their expectations are exceeded. The result is that the vast majority of satisfied customers whose expectations were met don’t feel inclined to write an unsolicited review.
This has two impacts on Yelp reviews:
- Satisfied customers are under-represented on Yelp
- The bulk of reviews are from “lovers and haters” (i.e. polarized 5-star and 1-star reviews)
In fact, reviews have only become more polarized with time. The distribution of 5-star reviews on Yelp [3] has grown 31% over a 5-year period (from 32% in 2009, to 42% in 2014). And the distribution of 1-star reviews has grown a dramatic 86% during the same period (from 7% in 2009, to 13% in 2014).
Does the Yelp filter introduce bias?
Yelp “filters” reviews they feel are potentially fake or unreliable according to an algorithm, and buries them in a section called “reviews that are not currently recommended”.
Often 25% or more of all reviews are filtered, including many legitimate reviews from actual customers. Although Yelp doesn’t reveal their criteria for filtering, anecdotally when a customer has only written a few reviews on Yelp, their reviews are filtered – Yelp is presumably looking for a history of reviews before they will consider a person’s reviews “recommended”. It is possible that Yelp’s review filter is also meant to motivate new reviewers to write more reviews, connect with friends, and to engage with Yelp as a means to make their reviews visible.
Yelp often doesn’t recommend reviews that are over-enthusiastic or especially harsh in their language, and hides these as “not recommended”. Many a passionate reviewer will inadvertently use hyperbole, exclamation marks or uppercase text without realizing that their review might now end up in the filter of “reviews that are not currently recommended”.
Ultimately, the Yelp filter exacerbates the problem of sample size. When reviews are filtered, there are even fewer reviews (i.e. smaller sample size) available among the “recommended reviews”, leading to a mathematically less accurate rating of a business.
And, yes, the filter introduces bias by removing legitimate reviews.
Does asking for reviews introduce bias?
Yelp goes so far as to explicitly tell business owners “Don’t ask for reviews“. Yelp argues that a business owner will only ask their happy customers for reviews, and that these requests for reviews will introduce bias.
Well, yes and no. Yes, of course, business owners will ask their happy customers. Yes, this initially introduces bias. No, the higher volume of reviews that results from solicited reviews will actually create for a more representative – statistically more accurate – star rating.
And when customers who had a negative experience see that the same business has a high rating, they are more motivated to share their contradictory experience. So the reviews from happy customers will attract reviews from unhappy customers, and over time an accurate equilibrium of ratings can be reached as a business collects more reviews.
Which leads us to the next question…
To ask or not to ask for reviews?
We take the position that solicited reviews create for a more accurate picture of a business, and actually benefit both businesses and consumers.
If business owners could ask for reviews, the business ratings might actually be more accurate and less biased:
- Business owners usually ask their happy customers for reviews (sampling bias)
- Reviews from happy customers attract unsolicited reviews from unhappy customers (reduction of sampling bias)
- More reviews ultimately result in a more statistically accurate star rating (increased accuracy)
Whereas, if a small business owner or professional does NOT ask for reviews (i.e. Yelp’s policy), inaccuracy and bias are introduced:
- we have the statistical problem of insufficient sample size (reduced accuracy)
- which is then exacerbated by Yelp’s review filter that removes legitimate reviews (bias), and further reduces the sample size (reduced accuracy)
- and the satisfied majority may be under-represented (expectation bias)
Asking for client reviews still assumes that there is a process that filters fake and inappropriate reviews using a protocol that looks at review language and other patterns. This process would allow would-be reviewers to write as few as one review without their review being filtered.
Summary
Yelp may not be illegal, but the combination of the Yelp review filter, Yelp’s policy against asking for reviews, and the fact that most satisfied consumers don’t voluntarily write reviews, stacks the deck against small businesses and professionals, and is a disservice to consumers when real customer reviews are “not recommended”.
Yelp business ratings favor businesses with more customers like restaurants and retail, and are less representative of small businesses and professionals with fewer customers like contractors, attorneys, real estate agents, advisors, and consultants.
Although reviews are valuable, remember to take business ratings with a grain – or spoon – of salt if a business only has a few reviews.
Looking for a Yelp alternative? Real Word of Mouth not only addresses the issues of bias due to Yelp policies and the Yelp review filter, but we address many other Yelp deficits when it comes to online business marketing. Check us out!
Addendum
For those reading this with a background in statistics, here is a short list of some of the shortcomings in the statistics calculations used in this article:
- Review distribution is not Gaussian
- Reviews are discrete and not continuous
- Reviews are influenced by previous reviews
References
1. ^ “FTC subpoena revelations, thousands of complaints send Yelp’s stock price tumbling” by Neil Ungerleider, posted on Fast Company blog on 4 April 2014, retrieved 14 October 2015
2. ^ “Levitt vs Yelp! Inc.”, No. 11-17676, United States Court of Appeals for the Ninth Circuit, opinion by Judge Marsha S. Berzon, filed 2 September 2014.
3. ^ “Reviewers Mostly Positive, 67% on Yelp Are Either 4 or 5 Stars” by Greg Sterling, posted on Screenwerk.com blog on 19 November 2014, retrieved 14 October 2015