Test the limit

Most Top 500 retailers testing and monitoring their e-commerce sites use one of three vendors. With subtle differences between them, it’s important to find the right fit.

Zak Stambor

Things Remembered Inc. had tested its e-commerce site in advance of peak periods, but still suffered several outages when traffic ramped up a little more than three years ago. The problem, the retailer's chief information officer Mark Lilien found, was that the vendor testing the site wasn't able to adequately replicate the complex, slightly chaotic, and often convoluted ways that shoppers browse and buy on ThingsRemembered.com.

"The way our customers shop isn't linear," Lilien says. "We needed to be testing based on the actual ways that shoppers shop and orders are processed." That proved difficult on the site because each of the retailer's roughly 4,000 SKUs can be personalized in several ways. For example, a shopper can have one corner of a fleece baby blanket embroidered with up to two lines of text in any of 10 colors, while a pair of wine glasses can be engraved on both its front and back with up to five lines engraved in eight colors. And about 40% of its web site visitors arrive at the site via a web search, according to Internet Retailer's Top500Guide.com—and many of those are searching for a specific item—which means they start browsing at a product or category page rather than the site's home page.

It was a complex but important problem for the retailer, which booked $38.9 million in web sales in 2012 and operates more than 600 bricks-and-mortar stores. The retailer chose Compuware APM's web load testing tool because the vendor uses statistical probability to test ThingsRemembered.com's limits in a way that mirrors the many ways visitors use its site. For instance, if 10% of online shoppers have their items gift wrapped, the load testing—which generates a high volume of traffic from multiple locations to see if the site's delivery chain can handle the stress—mirrors that pattern. After the test, the vendor presents the retailer with a report detailing any infrastructure or software component that is failing under the pressure of high traffic.

Within a half hour of running its first Compuware load test, Things Remembered uncovered several hardware and software issues it had to address, such as the need to add server capacity.

Things Remembered also began working with Compuware APM's web monitoring tool, a feature the vendor added following its 2009 acquisition of performance-monitoring company Gomez, to track the site's performance across the country. Compuware's combination of testing and monitoring tools is one reason the retailer chose to work with the vendor.

By knowing what a site is capable of, a retailer like Things Remembered can find and resolve potential issues before shoppers find them, and be ready to quickly respond if the site slows down. While performance-monitoring services can be costly, the cost of failing to be proactive can be even costlier, experts say. And a service that suits one web retailer might not be best for another.

Research firm Gartner Inc. positioned Compuware APM as a "leader" in its "Magic Quadrant for Application Performance Monitoring," but it was far from alone. In fact, Gartner research vice president Jonah Kowall, the report's author, wrote in a blog post, "Everyone has varying application architectures, hence oftentimes little or unknown APM vendors might be the best fit for your environment."

That explains why among the 500 merchants in Internet Retailer's 2013 Top 500 Guide, there are several dozen companies listed as web site performance monitoring vendors, many of which have only a handful of clients. But three vendors—Compuware Corp., Keynote Systems Inc. and SmartBear Software's AlertSite—account for 236 of the retailers in the guide and a big majority of those that named a site performance vendor. (Among the rest, 154 use an in-house system in concert with, or in lieu of a vendor, and 107 declined to name their vendors.) And those three vendors also hold the top three spots in Internet Retailer's Top Tech 2014, which ranks vendors by the combined web sales of their clients in the Top 1000.

Each retailer has to determine its site's needs to find the best fit, says Peter Sheldon, principal analyst, Forrester Research Inc. For example, if a retailer has a robust mobile app, it might turn to Keynote because of its expertise at using mobile devices to monitor apps. And while different retailers may find different answers, "every retailer needs to be concerned about testing," he says.

The three leading vendors have similar, but slightly different offerings (while starting pricing is noted, it is highly variable, says Sheldon, because of variations including the scope of the tests and monitoring, the number of sites and the volume required):

For Things Remembered, whose multiplicity of options make simulating visitor patterns difficult, Compuware has proven a good choice, Lilien says.

When Things Remembered makes a change to its site, Lilien uses the vendor's self-service tools to run a load test that stresses the site to see whether the move will cause problems. "We have to know what our site can handle," he says. "Otherwise we're too late."

For example, the personalized gifts retailer last April gave shoppers the option to buy an engraved item online and have it waiting for them in a store four hours later. And while it tracked the percentage of consumers who selected the option during its busy seasons—Mother's Day, Father's Day and graduation—it remained uncertain how the site would respond during the hectic winter holiday rush.

Offering buy online, pickup in store in combination with personalization can tax the site, Lilien says. That's because the site has to recognize the items' idiosyncrasies, such as where they can be engraved, and check to determine which items are in stock at a particular location.

To avoid potential hiccups during the holiday rush, the retailer used Compuware's tool to run load tests about eight weeks before the start of the holiday season. In preparation for the tests Things Remembered forecast how many shoppers it expected on its site—it had an average of 990,000 monthly unique visitors in 2012, according to Top500Guide.com—and diagrammed about 30 different paths those shoppers were most likely to take, including those who would select the buy online, pickup in store option. The retailer then tested what would happen if more than 10 times more consumers visited the site than the retailer anticipated. It ran the test at 3 a.m. so that if the test traffic brought down the site few, if any, shoppers would be impacted.

The test helped the retailer find and fix weak spots, such as the site making unnecessary database calls. Those fixes were completed before November, helping the retailer speed up its site and avoid any significant downtime during the holiday season. Moreover, that performance likely contributed to the retailer setting sales records, Lilien says.

Knowing potential weak spots has helped The Jones Group Inc. create contingency plans in case any of its sites slow down, says Michael Hines, vice president of mobile strategy and e-commerce systems for the retailer, which operates several retail sites for brands including Jones New York, Anne Klein and Nine West. The retailer eight years ago began working with AlertSite because the vendor made it easy to set up the click path for both load testing and ongoing site monitoring.

Before last holiday season it used five of its most common click paths to run load tests that simulated the impact of traffic four times the number of typical visitors during the holiday season (which is roughly double its sites' non-holiday traffic). It discovered that the content on category pages on its sites like NineWest.com— multiple images, brief product descriptions, available colors and the price—was slowing the site down when it came under what Hines calls "extreme duress."

After identifying that content as a potential problem, Hines and his colleagues developed a plan to manually tweak the site's coding if it received an automated call from AlertSite notifying it of a site slowdown. The planned change was to present a fraction of the products on the category page. For example, rather than show a shopper all 91 pumps on NineWest.com, it would show her a smaller number, say 20 pairs, and require her to click through pages to see all of the items in the category.

"By making that small change, we'd avoid having an unusable site experience," Hines says. "Instead we'd be giving shoppers a reasonably good site experience." Even though the site didn't slow down to the extent that it needed to make the move during the holiday season, Hines says the planning was worthwhile because it was armed with a plan to avoid losing revenue.

AlertSite also helps The Jones Group provide an experience superior to its competitors, he says. The retailer works with AlertSite to regularly track load times for five of its competitors—both those on the same Demandware e-commerce platform that it uses, and those using other technology. Benchmarking against other sites, particularly those on the same platform, lets the retailer identify issues that it might not otherwise have uncovered. For instance, if the retailer finds that a product page on another site loads 700 milliseconds quicker than a comparable page on NineWest.com, it can dig into the features on its page to see if an element like multiple product views is slowing the page down. It can then adjust. For example, instead of loading alternate images before the site fully displays, it might load them after the main components on the page load.

"Over time if you don't pay attention, you can find that small additions to a page can make a site that used to be fast much slower," Hines says. "The only way to combat that is to periodically review our performance."

Online ticket marketplace StubHub, a unit of eBay Inc., doesn't want to wait to unearth any slowdown; it wants to know about any delay right away before shoppers have problems making purchases. That's why it works with three monitoring vendors—including Keynote and AlertSite—to closely track its site's performance, says Hanna Sicker, the company's site operations manager.

"If you only use one monitoring tool you might be missing something," she says. "There are so many possible users that you have to cover your bases."

Keynote, for example, might catch something that AlertSite misses, she says. Or, if all three vendors sound alerts, the marketplace can piece the information each provides together to quickly identify and resolve the problem. The different vendors provide different information. Keynote makes clear whether the problem is with the site's server, network, application or database; the marketplace's AlertSite tool reports that the script failed but doesn't provide much detail; and another service identifies the general area where the issue arose, she says. And even though AlertSite provides the least detail, it is usually the first alert StubHub receives.

Because Keynote provides the richest detail, StubHub uses it to run about 100 scripts to ensure that the site is loading quickly, while it runs only a handful of its key site elements with the other vendors.

It also uses Keynote to test its mobile site and apps, which launched last August. Mobile is increasingly important for StubHub because it accounted for about half of its traffic last year and about 28% of revenue. But because mobile app monitoring requires using actual mobile devices to monitor performance, it is more expensive than tracking a desktop site, which is why StubHub only uses Keynote to track its mobile performance.

Using several tools in concert has helped keep the online marketplace humming; its site outage time was about 50 minutes in 2013, down from 96 hours two years earlier. That's largely due to the site focusing on performance and availability.

"If someone can't see the ticket on our site because the site isn't loading properly, they aren't going to buy it," Sicker says. Instead, they're clicking elsewhere, leaving the site without making a purchase. That's money left on the table. Finding the right vendor to test and monitor its site can help a retailer find and fix issues before they blow up.

For Things Remembered, those blow-ups were extremely costly. Since it began working with Compuware it has largely avoided outages. And while the vendor's cost is "unreasonably high," Lilien says, its "value is even higher."




application monitoring tools, February 2014 magazine, Load testing, Performance management, web load testing tool