Changes announced today to Gartner Peer Insights – effective January 31 – underline how the customer reviews site has grown up, and introduce a new strategy from Gartner. But they also signal the start of a new crackdown on vendors, in particular to discourage – and possibly penalize – the cherrypicking of top references.
Peer Insights has been elevated by Gartner management to the “trifecta” of key research, alongside Magic Quadrants and Critical Capabilities. Its development is now moving to a more standardized, predictable pattern of quarterly updates, coupled with six-monthly updates to the Customers’ Choice methodology.
The first wave of changes are both a move to extend the appeal of Peer Insights to a wider base of users – and to clamp down on vendors said to be “gaming” the reviews process. For example, Gartner cited examples of vendors sending out an NPS survey, and then only inviting top responding customers to submit reviews.
For most vendors, there is a minor difference in average scores from Peer Insights reviews generated via vendor-driven programs, and those coming in “over the transom” from enterprises. However, in a briefing earlier this week, Gartner GVP Richard Cho told me there are “very large” differentials in scores for certain vendors between vendor-sourced and organic reviews. “We believe they are actively trying to cherrypick the best customers,” he added. “Publishing overall averages will discourage gaming.”
Gartner has also identified that vendors who have notched up more than 50 reviews tend to get complacent in driving fresh reviews – especially if they have a high average score.
But policing reviews will always be an arms race, and getting to the truth is hard. Just as vendors strive to determine the real influence of industry analysts in major deals, so Gartner is targeting vendors who might be a bit “over-enthusiastic” in persuading selected customers in providing reviews. There’s a long way to go, since today, 9 in 10 reviews come via vendor campaigns. Previously, Gartner has offered limited numbers of $25 gift vouchers for reviews generated from special URLs provided to vendors to share with their customers.
In seeking to tighten control, Gartner is copying rival reviews site TrustRadius, which already gives higher weighting to reviews sourced directly. And to remove any remaining doubt as to how seriously Gartner is taking the task of weeding out over-favorable reviews, Cho warns: “We will catch those who try to cheat, and we will discipline them.” It’s not the first time Cho has chosen such emotive language – last April, he caused uproar at KCG Connects for threatening AR pros that Gartner would “punish” vendors caught trying to influence reviews.
Requesting reviews is a double-edged sword
However, verifying user identities is always a contentious issue. According to Brandon Davis, Senior Manager, Analyst & influencer Relations at Spredfast + Lithium, many users are uncomfortable about providing their work email address to Gartner or other reviews sites. The convenient option of signing in with LinkedIn, comments Davis, is also proving unpopular since it defaults to agreeing to let Gartner Peer Insights store LinkedIn profiles. To make it worse, says Davis, however you have logged in, “the sites then periodically request reviews for a multitude of other software companies”.
His firm recommends that reviewers opt out of future communications from Gartner, when providing a review – to avoid further requests, and to allay fears that Gartner is slowly but surely compiling a comprehensive list of contacts to sit alongside its existing data on who buys what from whom, and how much they pay for it.
Davis is also mindful that customers have only a limited amount of time and willingness to provide reviews for their suppliers. He says: “By taking care of the customers you’re asking to write reviews and creating a positive experience, they are more likely to participate in other advocacy activities. However, if they get burned doing you a favor, you’ve killed the future content potential for that customer.”
Peer Insights introduces role-based reviews
Conscious that its 170,000 Peer insights reviews are largely (85 percent) from enterprise buyers – defined by Gartner as companies with annual revenues of more than $50m – Peer Insights is also introducing role-based surveys. II doing so, Gartner is eyeing the extensive but as-yet-untapped market for reviews from “business or application users who don’t have a lot of info on the purchase of the software but have a lot of experience using it”. Cho adds: “Getting the voice of those application users is exceedingly helpful.”
To accommodate this, a new front end for submitting Peer Insights reviews will ask the reviewer for their role, and then individual users will be directed to a truncated set of questions.
Cho says: “Getting the voice of the end user is critically important for buyers.” He believes today’s changes will “provide the right incentives for vendors not to cherry pick but to get all customers to write reviews”: something that has been on Gartner’s wish-list for at least a year.
Gartner also expects a surge in traffic – with Peer Insights hits already “far surpassing” the Gartner.com landing page – for another new feature: feature comparisons between vendors, based on review data. Each review has “hundreds” of data points, according to Cho.
Bi-annual methodology updates and quarterly feature updates
One element of Peer Insights that has so far frustrated marketers and AR professionals alike has been how new features have been introduced without fanfare, while review methodologies have changed without warning. Gartner wants to change that and is introducing bi-annual methodology updates, coupled with feature updates on a quarterly basis. More insights into Gartner methodology are also promised for marketers and ARs.
Due to launch later this year, the next step in Gartner’s plan for Peer Insights will be the planned introduction of the first local language sites – probably Spanish and German, plus possibly Chinese – using (we believe) machine translation to localize content (how to navigate the perils of machine-translated content is a story for another day).
Our take: Beware privacy issues and watch out for customer fatigue
With the latest updates, the Gartner reviews site is coming of age. With a claimed 170k reviews, Peer Insights is no longer engaged in an arms race for review inventory against TrustRadius and G2. In fact, Gartner is now attempting to move the goalposts, not just for Peer Insights, but also for rivals. As Cho commented: “Our competitors monetize their traffic, we monetize our content.”
It’s clear that Gartner is preparing for a new battle – this time against vendors. Expect that reviews generated via vendors’ own marketing campaigns will be treated with suspicion by Gartner.
And in its ongoing attempts to filter out bias, Gartner again claimed this week that the decisive factor in reviews is the sheer volume. The introduction of role-based reviews is clear evidence of a further move in that direction – opening the door to multiple reviews for the same vendor, from the same customer.
However, privacy may still prove to be a major stumbling block: even vendors with mature peer review programs are finding that customers are unwilling to personally disclose their information to a third party – especially, it seems, when it comes to verification via LinkedIn accounts.
We also think that review fatigue will become an increasing problem – with a possible backlash from customers against vendors who ask for too many reviews, too often.
If you have ever stopped filling in a market research survey mid-way after becoming uncomfortable with the amount of data you’re being asked to disclose, then you will know the feeling that reference customers will experience in 2019.
Not only are there three major sites (which one to choose?), but also there’s little – if nothing – in it for the customer, especially as review volumes increase, and as your rating becomes just one in 100,000 – except perhaps getting bombarded with follow-up emails.
Would you like to drill further into this? Click here to get in touch!
