Two years ago, Destrier’s Playbook highlighted how vendors were ‘weaponizing’ enterprise peer reviews – for example, to counter a low position in a Magic Quadrant or to highlight their strengths and exploit competitor weaknesses.

Now a new use case is emerging – vendors using Gartner Peer Insights data to put pressure on Gartner’s analysts, especially in the always emotive instance of arguing about scores in a pre-publication Magic Quadrant.

Often a source of frustration for vendors and analysts alike, the failure to reach common ground on scoring – or a vendor’s omission from a report – can lead to delays in the publication of a Magic Quadrant.

Vendors and analysts go head-to-head on evaluation report scoring

Historically, the process for any MQ, Forrester Wave, or other competitive market evaluation is that vendors argue that their scoring looks relatively low in selected areas. The analysts then review and sometimes revise the scores – but often, not to the level where the vendor had indicated they thought they merited.

In Gartner’s case, this is where the Ombudsman eventually steps in – although there are several preliminary steps on the escalation ladder.

Usually, the analyst firms will stand behind their score. They’ll tell the vendor that the lowball scoring is justified because of what they have observed – such as inquiry data and secondary market research.

Of course, all of this data is inaccessible to vendors. There’s very little the vendor can do on this uneven playing field.

Occasionally, a vendor goes nuclear

Once in a blue moon, the dispute between vendor and analyst boils over into legal proceedings – but even then, the vendor is on shaky ground: The ultimate defense for an analyst firm is that scoring is their “opinion”. Therefore, in line with the US First Amendment, and the Universal Declaration of Human Rights, upheld in most civilized countries, it’s free speech – and there’s not a lot a vendor can do – at least, not related to the report itself (More drastic measures, like canceling multi-million dollar contracts, are sometimes threatened).

Now, vendors are trying to turn the tables by pulling Gartner’s Peer Insights data to show analysts that their draft MQ scoring is out of line. This new approach gives vendors access to some of the same real-world data that analysts have long relied on. A vendor that’s rated lower than competitors for particular capabilities can highlight how, according to GPI, this is not the case.

Gartner Peer Insights chief Anatoli Olkhovets points out there may be “legitimate reasons” why a vendor is highly rated on Peer Insights, but still a challenger on the Magic Quadrant – as analysts draw from a more comprehensive selection of data sources than just GPI – and that customers tend to only use one of a number of competing products or services.

“The analysts are looking at sales execution, pricing and market presence, system integrator presence and market understanding, as well as the speed of a vendor’s market innovation,” he comments, adding that smaller, up-and-coming vendors tend to focus on GPI as they must “thrive to survive” – and just equaling the big, established players on GPI scores isn’t enough.

Our take: Data from GPI is undoubtedly making it clear to analysts that they need to be more accountable and transparent in vendor scoring for Magic Quadrants. Although we still see instances where analysts are dismissive of GPI, the voice of the customer data it surfaces isn’t going away: GPI now has 415,000-plus reviews. What’s more, it’s expanding beyond the markets covered by MQs and Market Guides.

Of course, to use GPI data in negotiating with Gartner analysts, vendors still need review inventory: the more, the better. And once reviews nudge into the hundreds, then the voice of the customer is loud enough to influence Gartner’s analysts more than they would care to admit.

If you’d like to give a bit of pep to your peer reviews program, talk to us!