Most Top 500 retailers testing and monitoring their e-commerce sites use one of three vendors. With subtle differences between them, it’s important to find the right fit.
Things Remembered Inc. had tested its e-commerce site in advance of peak periods, but still suffered several outages when traffic ramped up a little more than three years ago. The problem, the retailer's chief information officer Mark Lilien found, was that the vendor testing the site wasn't able to adequately replicate the complex, slightly chaotic, and often convoluted ways that shoppers browse and buy on ThingsRemembered.com.
"The way our customers shop isn't linear," Lilien says. "We needed to be testing based on the actual ways that shoppers shop and orders are processed." That proved difficult on the site because each of the retailer's roughly 4,000 SKUs can be personalized in several ways. For example, a shopper can have one corner of a fleece baby blanket embroidered with up to two lines of text in any of 10 colors, while a pair of wine glasses can be engraved on both its front and back with up to five lines engraved in eight colors. And about 40% of its web site visitors arrive at the site via a web search, according to Internet Retailer's Top500Guide.com—and many of those are searching for a specific item—which means they start browsing at a product or category page rather than the site's home page.
It was a complex but important problem for the retailer, which booked $38.9 million in web sales in 2012 and operates more than 600 bricks-and-mortar stores. The retailer chose Compuware APM's web load testing tool because the vendor uses statistical probability to test ThingsRemembered.com's limits in a way that mirrors the many ways visitors use its site. For instance, if 10% of online shoppers have their items gift wrapped, the load testing—which generates a high volume of traffic from multiple locations to see if the site's delivery chain can handle the stress—mirrors that pattern. After the test, the vendor presents the retailer with a report detailing any infrastructure or software component that is failing under the pressure of high traffic.
Within a half hour of running its first Compuware load test, Things Remembered uncovered several hardware and software issues it had to address, such as the need to add server capacity.
Things Remembered also began working with Compuware APM's web monitoring tool, a feature the vendor added following its 2009 acquisition of performance-monitoring company Gomez, to track the site's performance across the country. Compuware's combination of testing and monitoring tools is one reason the retailer chose to work with the vendor.
By knowing what a site is capable of, a retailer like Things Remembered can find and resolve potential issues before shoppers find them, and be ready to quickly respond if the site slows down. While performance-monitoring services can be costly, the cost of failing to be proactive can be even costlier, experts say. And a service that suits one web retailer might not be best for another.
Research firm Gartner Inc. positioned Compuware APM as a "leader" in its "Magic Quadrant for Application Performance Monitoring," but it was far from alone. In fact, Gartner research vice president Jonah Kowall, the report's author, wrote in a blog post, "Everyone has varying application architectures, hence oftentimes little or unknown APM vendors might be the best fit for your environment."
That explains why among the 500 merchants in Internet Retailer's 2013 Top 500 Guide, there are several dozen companies listed as web site performance monitoring vendors, many of which have only a handful of clients. But three vendors—Compuware Corp., Keynote Systems Inc. and SmartBear Software's AlertSite—account for 236 of the retailers in the guide and a big majority of those that named a site performance vendor. (Among the rest, 154 use an in-house system in concert with, or in lieu of a vendor, and 107 declined to name their vendors.) And those three vendors also hold the top three spots in Internet Retailer's Top Tech 2014, which ranks vendors by the combined web sales of their clients in the Top 1000.
Each retailer has to determine its site's needs to find the best fit, says Peter Sheldon, principal analyst, Forrester Research Inc. For example, if a retailer has a robust mobile app, it might turn to Keynote because of its expertise at using mobile devices to monitor apps. And while different retailers may find different answers, "every retailer needs to be concerned about testing," he says.
The three leading vendors have similar, but slightly different offerings (while starting pricing is noted, it is highly variable, says Sheldon, because of variations including the scope of the tests and monitoring, the number of sites and the volume required):
- Compuware's load testing tool enables retailers to test their sites using real web browsers, such as Internet Explorer, Firefox, Chrome and Safari, and/or emulated browsers (emulated browsers mimic how a web page will render, letting a retailer test how its web pages will perform with multiple browsers without having all those browsers and versions installed on a computer). It can test how a site will perform if more than 1 million users attempt to access it at the same time and its application monitoring tools identify the source of performance problems. Compuware's load testing option starts at $5,000 a year and a bundle that includes consulting services and continuous monitoring, starts at $20,000.
- Keynote's load testing tool lets retailers use either real or emulated browsers to test the impact of up to 40,000 concurrent users at the same time for those using its tool to run tests on their own and more than 1 million users for those working with the vendor to run tests. It can track how a site performs on Internet Explorer and Firefox, and how sites and apps appear on mobile devices from more than 275 locations worldwide. Its load testing tool starts at $300 a year and its monitoring tools start at $200 a month.
- AlertSite's load testing tool lets retailers use emulated browsers to examine how shoppers see the site on Internet Explorer, Firefox and Chrome. It offers a range of monitoring options—for instance, a retailer can monitor its third-party content, such as a vendor-hosted ratings and reviews tool, to identify what is causing a site to load slowly. Its most basic package, which monitors one web site's availability every five minutes, starts at $12 month, and the price tag goes up as retailers add additional features.