Some recent data out analyzing online reviews on sites like Amazon and Yelp show that half of all reviews give the highest possible rating possible. So, ask yourself, of all the stuff you bought on Amazon and all the restaurants you tried on from Yelp, would you give the highest rating possible?
Human nature would tell us of course not! This phenomenon is known as a “Positivity Problem“. It’s like when I asked you all to give me a 5-star review on my book “The Talent Fix” and you all did! (Well, all except like two of you, and the next time I see my Dad I’m really going to give him hell over that 3-star review!).
Basically, “star” ratings are unreliable rating mechanisms.
The big problem is when we measure things like employee experience, candidate experience, etc., we use star rating measurements. Glassdoor, the employer review site, turned job board, gained millions of followers and giant traffic by using anonymous star reviews from employees which were almost immediately problematic, but we all ignored it and took it as the word of God. OMG! Amazon is a 5-star employer and Walmart is a 2.5-star employer. Both of which are most likely wrong.
What should we be doing to find an accurate measure?
Like any great leader, we should be looking less at a number rating and more at the verbatim comments. If you have a giant sample size and can use AI to gain further insights, you can really gain some understanding of what candidates and employees are thinking, or which employers truly are the best or worse to work for.
The study mentioned above found that the comments using the most emotionally charged wording, both positive and negative, were greater predictors of measuring success. Comments become a great predictor of what’s really going on.
What can go wrong with this? I’ll call this the “Business Insider” Dilemma!
The BI Dilemma is a phenomenon I’ve noticed recently with writers at Business Insider. I’ve been a long-time reader of BI and they produce some really good content, but over the past 12-24 months they tend to make some extremely bold conclusions based on comments from only a handful of employee’s comments.
Things like, Amazon is an awful employer for women, just come and read about twelve women we spoke to who are former Amazon employees, out of 300,000 Amazon employees who are female! Come on!
To be fair, BI isn’t the only media outlet to do this, it’s become common place in cancel culture. Find a minority opinion and run with it as a majority opinion. This is a problem when it comes to using employee and candidate comments as a measure of success or failure.
That number has to be pretty substantial to gain real insight. At a minimum, I would think you would need about 5-10% of all to comment to be comfortable in making any real decisions based on comment data. So, if you have one thousand employees, I want to see about 100 comments to get a true feel for what’s really going on.
We focus so heavily right now on “data” on the number and the movement of the number. But if we know that star ratings alone are unreliable, do we truly feel like we are getting the full picture of what’s really needs to improve or change?