Gomez launches new benchmark index to measure site consistency

The average speed with which a shopper completes a retail transaction on a site and the average availability of web pages are key metrics used by online retailers to rate the performance of their web sites against those of their peers – but they don’t tell the whole story. The degree to which delivery of those services is consistent – or not – across time and traffic peaks can isolate performance issues in greater detail than do average response time and availability, on their own. That’s the rationale behind a new index from web performance monitoring provider Gomez inc., the Gomez Consistency Index, which launches today.

The new index offers online companies the ability to understand the consistency – with a score that blends two metrics, reliability and variability – of their Internet service deliverability. By focusing on web site consistency, according to Gomez, companies can fine-tune their applications, processes and infrastructure to achieve their Internet performance objectives. The Consistency Index benchmarks monthly consistency scores calculated for 15 top online retailers.

Information on the average performance of a web site – how many seconds it takes on average in the course of a day to deliver a retail transaction – provides a single value, but provides no insight on how the web site performs at specific times – for instance, during high holiday traffic or right after a major online promotion, says Bill Betcher, Gomez product manager. “You don’t see the fluctuations and how well your infrastructure responds to customers at the time they want to buy most, which is when you have the most customers online,” he says.

The new index addresses that issue with the consistency score, which blends information on variability and reliability of site performance. The variability element maps out peak load times and how the web site performs against them. The reliability component represents actual outage in minutes per monitoring period. Gomez’s new index will publish separate benchmarked scores for reliability and variability, as well as the consistency score, a blend of the two metrics.

One practical value of the consistency score, in combination with the average performance and availability score in Gomez’s other index, is that it provides information to inform IT spending. For example, CDW.com, part of Gomez’s original performance and availability index since January, averages about 21 seconds to complete a transaction, placing it in the bottom half of the 15 indexed retailers on that measure.

In the just-published consistency benchmark index, however, CDW ranks number one. “They’re the most consistent retailer that we monitor,” says David Flinn, director of performance excellence at Gomez. “Just because your site is slower it does not mean you are ‘worse,’” he says. For instance, if a company is spending less on IT infrastructure, and its site is slower than its competitors’ but it has same consistency score, “It’s doing pretty well,” says Flinn, “If you’re already as consistent, maybe you don’t need to spend more. Maybe you can deploy that money into other business functionalities. It gives the retailer another way to decide.”

For the reporting period of August 1 through August 15, the average consistency score for the 15 retailers indexed was 75.3. Under the score, which Flinn compares to the mathematical concept of standard deviations, CDW, with the lowest score, 12.2, ranked the highest. The retailers’ web sites ranked as follows:

CDW 12.2
Amazon 15.4
Office Depot 15.8
Gap Online 18.4
Wal-Mart 26.6
Office Max 31.6
Sears 34.5
Target 48.6
Best Buy 60.7
J.C. Penney 67.9
Average 75.3
HP 80.5
Lands’ End 89.9
Staples 166.5
Sony 224.1
Dell 237.2.