Last week we received a comment from a reader questioning the reliability of the Keynote Mobile Commerce Performance Index. Mobile and web performance management firm Keynote Systems Inc. provides this index exclusively to Internet Retailer every week. The index measures m-commerce home page site performance, both the load time (the amount of time it takes to download all elements of the page from the server to the smartphone) and the success rate (the percentage of times a page downloads successfully). The index is a great resource for retailers in m-commerce to use as a benchmark to guide their performance efforts and to learn from others.
The reader said the index is “in no way representative of the mobile retailer space.” Actually, Keynote Systems executives and I thoroughly scanned the mobile market to come up with retailers of different sizes (from Amazon.com Inc. to Toolfetch.com LLC), from different product categories (from J.C. Penney Co. Inc. to Victoria’s Secret to Walgreen Co.), and from different channels (from web-only merchant Overstock.com Inc. to TV and web retailer ShopNBC.com to retail chain Staples Inc.). This is an excellent cross-section of retailers, 30 in all, in the mobile space.
The reader went on to say that “some notable exceptions are missing from this list” and that “in the future I’d like to see a much more comprehensive list.” His letter was quite timely. Last week, after weeks of testing, we doubled the size of the index from 15 to 30. Again we added a good cross-section, including some more heavy hitters. As for “notable exceptions,” as with any list, people will have their own ideas of who they think should be on a list, and an index is designed to be representative, not all-inclusive. (The Dow Jones Industrial Average, for example, only tracks 30 stocks.) The 30 retailers on the weekly Keynote index really run the gamut of mobile commerce and show the range of site performance.
The reader wanted to know where the methodology was. It always is spelled out clearly in the last two paragraphs of the story. The reader then asked a technical question: “Android Chrome browser and Safari iOS have two different ways of rendering web pages, how can you ensure consistency in testing results?” Keynote responds, “We do not capture and include client rendering time in the index data because it varies not only because of different browsers but also because of physical characteristics and conditions of the device, such as CPU power, the number of background tasks, etc. By not including the rendering time, the index data reports the best load time possible for all sites equally.” In other words, the index doesn’t measure the time it takes the phone to render a mobile web page, thus avoiding the issue of how to measure rendering time consistently.
And finally the reader asks, “How can you compare a site like StrandBooks.com to a more full-featured mobile site like Amazon?” That’s part of the point of the list, to see how different sites and different approaches to m-commerce compare and contrast. This list isn’t just for smaller retailers, who are on a budget, or bigger retailers, who have millions to spend. This list is for everybody. A reader can look at the list and learn that a heavier home page can take more time to load; conversely, a lighter home page can take less time. A reader can look at those times and then pull up the sites on her smartphone and judge for herself how her m-commerce site should look compared with how much it should weigh in kilobytes. There are many lessons to be learned.
Keynote Systems is a highly reputable performance measurement and management firm that has worked with Internet Retailer to provide this excellent resource for readers. And for many more resources, click on the Trends + Data tab above. We’re here to serve you, the reader. And you can count on Internet Retailer to bring you the best information on e-commerce and m-commerce.