Yelping about Yelp

Hello, Cafe --

Back in Nov.-Dec. of 2010, I complained vigorously to Yelp's customer =

service dept. about what I see as a case of bad information design ... =

and when I felt like our discussion was going nowhere fast, I told =

them I'd move my yelping over to an international discussion list for =

ID professionals, where I knew I would get much better arguments & =

feedback, plus an opportunity to learn whether my negative reaction to =

the Yelp review model and website is peculiar to me, or is more =

broadly shared.

I've been putting off starting this discussion because I've been =

working even more intensely than usual, trying to launch 3 new =

websites (2 down ... one still to go! ;-), but decided it was time to =

make it a priority when yesterday I heard yet another related report =

on the US public radio program, _Marketplace_:

"The power of the online product review"
originally aired: 18 Jan. 2011
< =

http://marketplace.publicradio.org/display/web/2011/01/18/pm-the-power-of-t=
he-online-product-review/ =

 >

So, to get us started, here's my Yelp story....

My own troubles with Yelp began back in May 2010 when I set out to =

write an online review for the veterinarian my family has been going =

to for over 20 years. He had recently performed another outstanding =

surgery on my elderly dog (in this case, for a large mouth cancer =

which had ruptured), and after I finished telling him in detail how =

pleased we all were with the outcome, he asked me if I would consider =

writing an online review.

Ordinarily, I wouldn't do this (I have no time for, or interest in, =

writing product reviews), but this was an extraordinary occasion ... =

made even more so when I actually looked up his business online.

To quote from the Yelp review I later wrote for his veterinary practice,

	"I have a hard time recognizing the skilled and compassionate
	vet I've known for 2 decades in some of the negative reviews
	here and on other Web pages."

Not only were some of the reviews quite nasty little rants, but at =

Yelp, the negative reviews were given more prominence and weight. I =

really had to poke around to find the 5-star reviews -- almost all of =

which had been "filtered" (meaning, they are removed to a separate =

page, which you access via a link in small print at the bottom of the =

business's main Yelp page, after which you have to decipher a bunch of =

skewed type before the page with filtered reviews will display). =

If/when you make it this far, a notice in small gray type (easy to =

miss when you're scanning the Web page for the content you're after) =

informs you that

	"Note: The reviews below are not factored into the business's
	overall star rating."

Well, in my vet's case, this is kind of a big deal. There are (as of =

today) 6 reviews of his veterinary practice in this "filtered" =

category. Five of these (including my review) award him 5 stars; the =

other filtered review gives him 1 star. But none of these 6 reviews =

contribute to the rating his business gets at the top of the main =

page. What we see there is an overall 3-star rating (with an ad 2 =

inches below this for another area vet with a 4.5-star Yelp rating).

When I first wrote my Yelp review back in May 2010, I was concerned =

that my 5-star review might be filtered, as had so many others, so I =

read everything I could find about how to write a proper Yelp review, =

so that there would be no question about its legitimacy. And =

apparently there wasn't ... at first.

My review appeared right there at the top of his main page as soon as =

I posted it, so I thought I'd been successful, and moved on to other =

projects. What I didn't know was that Yelp's treatment of reviews is =

dynamic. Yelp actually alters the status of reviews *after* they're =

posted, without warning or notification.

I only discovered this when I returned to update my vet's review 6 =

months later, the day after my dog was euthanized.

At the time, emotions were pretty raw, and I was not pleased to =

discover that my review had been filtered and judged a "fake" by the =

omnipotent Yelp algorithm which reviews the reviews.

I hoped that updating my review would change my "fake" rating, but it =

didn't make any difference.

As a consequence, I wrote to Yelp to complain, and even this required =

a certain amount of ingenuity in order to bypass the (designed-in ?) =

limitations of their form for feedback. Among other things, I had to =

divide my comment into 3 parts, each of which was transmitted as its =

own response. Here's what I wrote in the 3 separate e-mail/forms:

1.
	(Part 1 of 3)
	My comments on improving Yelp all relate to the fact that my
	review of a local business has been "filtered" and made
	nearly inaccessible for reasons that are unclear to me,
	beyond the fact that I am not -- and have no intention of
	becoming -- an active Yelper. I was only persuaded to join
	Yelp in the first place because I felt that my 5-star
	experience and point of view was completely lacking from the
	reviews for the [business name] in [business location],
	posted to [URL for business's Yelp page]. Since my opinion
	still isn't properly represented, I see no reason to waste
	further time writing even more Yelp reviews that apparently
	don't count.

	I understand that there are major issues with people trying
	to game the system, and that Yelp has a genuine need to
	somehow filter reviews. But if my experience is any
	indicator, your filter does NOT do "a pretty good job" of
	this, as claimed in your FAQ.
	(continued)

2.
	(Part 2 of 3)
	I am a real, long-time customer of the [business name], and
	my review of the business is not a fake. When I wrote my
	review, I went to great pains to set up my Yelp account with
	an avatar, and a website address, which makes it quite clear
	that I am who I say I am. Plus, I was careful to write
	original and different reviews for the [business name] on
	Yelp vs. Google Maps, etc., so that my Yelp review of
	[business name] would not be filtered out because it was
	repetitive and/or available elsewhere.

	Moreover, I figured I'd been successful in this because my
	review originally appeared on the main Yelp page for
	[business name] directly after I posted it, and I was never
	informed that there was a chance it might be arbitrarily
	"filtered" out at some future point. It was only when I
	returned today to update my Yelp review of 6 months ago that
	I discovered it had been so filtered.
	(continued)

3.
	(Part 3 of 3)
	My actual complaint is not just that Yelp altered the status
	of my review without notifying me, but also that Yelp's
	ratings data pretends to be representative of all those who
	have yelped about a business, when in reality it is selective
	& skewed -- but without any warning that this is the case,
	and that some points of view have been mechanically censored
	using some mysterious algorithm which might very well result
	in mistaken exclusions. At the very least, there should be a
	decent-sized link to the filtered reviews at the bottom left
	of each Yelp page, accompanied by a brief warning that the
	above ratings are not compiled from ALL the reviews Yelp has
	received, but only from a selected few, with message along
	the lines of "click here to read the censored reviews".

	As it is, my limited experience with online review sites such
	as Yelp merely confirms all the negative criticisms of Yelp
	I've encountered elsewhere (e.g., =

http://marketplace.publicradio.org/display/web/2009/04/10/pm_yelp/ ).


This drew a response from Customer Service at Yelp directing me to a =

video explanation of their review filter at

http://officialblog.yelp.com/2010/03/yelp-review-filter-explained.html

which I found full of questionable assumptions.

I watched the video, and read all the related Web pages accessible =

from the above URL, and still didn't feel that my real issues had been =

addressed.

So, I replied with the following:

	[...] As far as I'm concerned, Yelp has every right to filter
	reviews ... and to even brag about the filter you've devised
	for doing so.

	But I think it is misleading (bordering on deceptive) not to
	properly warn people -- up front, in legible type, preferably
	right next to the starred rating -- that

	1. Yelp reviews are *mechanically* filtered.

	2. Filtering is dynamic, so that the exact same review may be
	judged "true" one day, and "fake" the next.

	3. A business's starred rating does NOT include any
	reviews which have been filtered.

	4. A business's starred rating changes NOT just because new
	reviews have been posted, but because previously "true"
	reviews may at any point be filtered out as "fake", and vice
	versa.

	Plus, I think this notice should include a clear, running
	tally of how many reviews (of the total received) were used
	to calculate a business's rating, and how many reviews were
	excluded from the ratings process (i.e., filtered).

	I would argue that this metadata is necessary because it's
	not just the Yelp cognoscenti who are reading (and probably
	relying on) Yelp reviews.

	Now that the search engines deliver up Yelp ratings to all
	sorts of people who (like me) know nothing whatsoever about
	Yelp and its unique ratings system, I believe Yelp has a
	responsibility to inform casual visitors that our
	expectations about how a starred rating is calculated (i.e.,
	the average of *all* received reviews) don't apply at Yelp.

	Had I known this from the beginning, I wouldn't be
	complaining now about Yelp's filtering mechanism having
	misjudged my review.

	In sum, I don't care how opaque your filtering process is, or
	what "trust factors" you use to assess reviews.

	What I do care about is that we're not told up front that
	reviews are mechanically filtered, with ratings skewed
	accordingly.

	I'm willing to bet that most people have no idea Yelp filters
	reviews -- or even that fake reviews are such a problem.

	I certainly didn't.

	Nor do I believe that Yelp's filtering mechanism -- or any
	filtering mechanism, for that matter -- can guarantee
	objectivity when it comes to assessing human motives.

	IMO, the best judge of human behavior (including deliberate
	deceit) is human intelligence, critically applied.

	Critical human judgment is not perfect, of course. But it
	works well enough for most of us to be able to maintain a
	satisfying social life, and even join with others in the
	pursuit of larger civic goals.

	Surely if we humans can manage this much, we can also figure
	out which online reviews to trust temporarily, until variable
	personal experience proves all involved either right or
	wrong. [...]

I probably should add here that there are a few things about the Yelp =

review model that I do like, such as the fact that Yelp allows =

reviewers to post updates (which I wasn't able to do elsewhere -- =

Google Maps, Bing Local, Kudzu), and also that they allow longer =

reviews -- 5000-character limit at Yelp, 4096-character limit at =

Google Maps, 4000-character limit at Yahoo Local, 2000-character limit =

at Bing Local (which also can't count properly! ;-), and a 250-word =

limit at Kudzu.

I should also say that, once I was able to access it, Yelp's customer =

service beat the pants off of that provided by Yahoo Local, which was =

so bizarre, it bordered on surreal, and I ended up feeling like I was =

trapped in some kind of endless computer loop ... although that would =

imply some sort of rationality that was completely missing from my =

Yahoo encounter! After 6 or so canned replies that made no sense, I =

grew so irritated with Yahoo customer service that by the time a =

thoughtful human being finally came to my aid, I had already =

proclaimed to all & sundry that I was done with all things Yahoo, and =

would NOT be posting my review to their site ... ever.

(And, of course, there's no going back on one's public ultimatums, no =

matter the cost, or there's no point in issuing them in the first =

place! ;-)

In sum, my little odyssey into the online world of product/service =

reviews left me questioning the whole genre.

I think there're a lot of problems with the *business* of online =

reviews, and what bugs me most about it, is the (sometimes extreme) =

partiality masquerading as universal truth.

Perhaps the behavioral economists have some answers.

I'm hoping some of our Caf=E9 mathematicians can explain the issues, =

too. Every teacher knows that our grading systems are always full of =

flaws, which are then rendered more or less moot by (predictable ?) =

human factors.

... Maybe we just trust to _The Wisdom of Crowds_ (yes, Surowiecki's =

book, but others toting the virtues of data mining for decision-making =

processes, too) to balance everything out.

But isn't group wisdom really a matter of scale?

When you only have 15 reviews of a business, with almost as many =

underlying value systems -- reviewers who give a 1-star rating because =

they think the color of the chairs in the waiting room sucks, casually =

intermingled with those who award 5 stars only when a vet raises their =

dog from the dead -- I'm not sure the mathematics of a simple, 5-star =

review model work well at all ... and especially not when websites =

like Yelp turn the 5-star rating system over to the machines....

Deborah
_____

Deborah Taylor-Pearce
[email protected]

___________________________________________________________________

Use the following address to post a message to all subscribers: =

 [email protected]

To subscribe, unsubscribe or change your options, visit:
 http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe

For all Information Design matters:
 http://InformationDesign.org

Problems? Write to:
 [email protected]
___________________________________________________________________