Beyond the Numbers

There is no longer any doubt that monitoring web site performance is key to delivering a satisfying shopping experience to customers. The only question today is: How do you find out what’s broken—and then how do you determine what’s important. With more consumers gravitating to the Internet to shop, and retailers allocating more of their IT budgets to create dynamic sites using rich media, retailers can no longer afford to fix every performance problem simply because the metrics indicate a site malfunction.

“One of the pitfalls of Internet retailing has been that performance benchmarks are not always chosen based on the performance criteria that are essential to the company’s overall health,” says Steve Weiskircher, senior director of information technology for electronics retailer Crutchfield Corp. “Performance metrics need to be prioritized according to what is key to a retailer’s business and achieving customer satisfaction.”

Take off the blinders

All too often, retailers look at performance metrics with blinders on, industry participants say. If a performance indicator falls below the prescribed benchmark, retailers typically rush to fix the problem regardless of whether it directly impacts revenue. As a result, costly fixes that don’t necessarily cause a customer to abandon a shopping cart or not return to the site can be made ahead of problems that do.

One explanation for why some retailers fail to prioritize properly is that they often find themselves awash in a sea of performance data gathered by third-party monitoring companies. Because they have not necessarily determined in advance which performance metrics are most critical to their business, they try to compensate by focusing on everything. The result is often a helter-skelter approach to correcting site performance problems.

“I have talked to retailers that spend weeks trying to fix the bells and whistles on their site that really don’t mean much to the customer, while other problems that may seem smaller, but do impact the shopping experience and revenues, go unattended,” says Heather Dougherty, senior retail analyst for Nielsen/NetRatings. “Retailers need to learn how to better parse and align performance data against the elements within the site that lead to higher conversion rates.”

In fact, consumers are willing to tolerate some site performance problems, such as a slow loading page, provided they can still complete the transaction they intend to make, site performance experts say. “If a hundred users are known to have encountered the problem and all of them complete a transaction, it is a good indication the problem is not a sticking point,” says Geoff Galat, vice president of product strategy for TeaLeaf Technology Inc., a San Francisco-based site monitoring service. “Before retailers can understand the importance of a problem, they need to drill down to determine the cause and the effect of it on their business.”

The first step to drilling down to identify the root of the problem begins with a retailer making the same level of commitment to the performance of a web site as it dies to the performance of its other sales channels.

Retailers who do not consider the web a major component of their business are more apt to unwittingly erect departmental silos that impede performance due to lack of inter-departmental communications. In other words, IT, merchandising, marketing and administration need to establish common performance goals that are understood by each department. “Prioritization of performance is an internal discipline,” declares Ron Rose, chief information officer for travel site Priceline.com.

Nailing the sporadic problem

One of the most common problems encountered by Internet retailers is the identification of a performance problem that shows up sporadically. While merchandising may have been made aware of the problem by a dissatisfied customer, the IT department may see only a performance metric within the accepted range and insist there is no problem.

In reality, the problem may occur with a subset of users, such as customers entering the site through dial-up connections. Consequently, the IT department is likely to miss seeing that a subset of dial-up users is encountering a performance problem if IT’s performance metrics capture sessions only from customers with broadband connections.

Worse, the problem can be isolated to customers with another set of common characteristics. Unless the IT department can be given a clear description of the problem, as opposed to being told “the page is loading slowly,” technical support staff goes by the numbers—and often those numbers don’t tell the story.

The caching problem

Other potential problems can be missed when a retailer pulls together many elements of a site in a customer’s web browser. The practice, known as caching, requires third parties to deliver each element to the shopper’s browser. Typically, third parties will deliver these elements using multiple nodes across the country to create a localized delivery system to expedite page download times. It is difficult and costly for retailers to monitor every node in use by the delivery partner, let alone isolate the node experiencing performance problems at the right time. Unless directly spotted, the problem will not necessarily show up on the IT department’s radar.

“If a potential red flag is considered okay from an IT perspective, some retailers are comfortable sticking their head in the sand than understanding if a problem is affecting key processes of the site for some customers,” says Mathew Poepsel, vice president and application manager for Gomez Inc., a Lexington, Mass.-based performance monitoring service. “A lot of times, problems have to be broken down before they can be properly identified.”

Key to breaking down the root cause of a performance problem is tying together the actual sales and customer activity data gathered on the site with the IT performance data. Doing so provides a broader view of the problem and allows retailers to identify customer clusters, such as those residing in a particular ZIP code or that are accessing the site from a particular link that may be experiencing a problem, but which shows up as a minor blip on the IT department’s radar.

“Allowing organizational silos to build up means that no single department really knows how all the performance and customer activity data gathered fits together,” explains Daniel Heimlich, vice president of marketing for Netuitive Inc., a Reston, Va.-based provider of performance monitoring software. “The data just sits in a vacuum and identifying the cause of the problem is like trying to find a needle in a haystack.”

Putting performance problems into context requires retailers to notify their performance monitoring services to focus on specific metrics and to produce reports on isolated incidents. “Once retailers educate their service provider about which performance metrics they care about, isolated incidents can be spotted and put into perspective against other isolated incidents that at first glance may not seem connected, but are actually creating a larger problem,” explains Alistair Croll, vice president of product management and co-founder of Coradiant Inc., a Poway, Calif.-based performance monitoring service. “The more angles from which a retailer can attack a performance problem, the better the chances of quickly determining its cause and impact on their business.”

Prioritizing the order in which performance problems need to be addressed starts with setting clear objectives. At the start of each quarter, Priceline.com gathers all departments to set marketing, merchandising, and performance issues for the next three to six months, and determine how best to achieve those goals. “Everyone knows what the goals are, then we figure out as a team the best path to march to in order to achieve them,” says Priceline.com’s Rose. “We have silos within the company, but communications between departments is not an issue.”

Measuring the suppliers

As an e-commerce portal that relies on third-party suppliers, such as airlines and hotel chains, to provide information, as well as links to their sites, Priceline.com measures performance data on its suppliers, as well as data on specific customer sessions. The data helps Priceline.com determine whether a problem resides within its infrastructure or that of a supplier. In the event a performance problem resides within a supplier’s site, Priceline.com will contact the supplier and provide details of the problem.

Priceline.com, which uses Keynote Systems Inc. and Netuitive to measure site performance, measures about 30,000 metrics on its site, including information requests of its database, the frequency of error messages, and CPU utilization throughout the day.

IRCE 2008 Conference Multi-Media CD-ROM

It is this kind of vigilance that keeps Priceline.com from tripping over itself when it comes to interpreting the significance of performance data. “The better a site monitors infrastructure, the better the chance of keeping a customer by not losing them to a performance problem,” says Rose.

Netuitive aids Priceline.com in this endeavor by capturing actual customer sessions where a performance problem has occurred, rather than relying on recreating the session. Replaying an actual session, which is time stamped, makes it possible to see the problem that occurred as the customer saw it, thereby giving the retailer a better sense of where the problem originated and its severity. At the same time, it resolves discrepancies between how IT and other departments view the problem.

Warning signs vs. trouble

“Some performance indicators are just warning signs as opposed to actual trouble,” adds Rose. “That’s why our performance metrics are prioritized and we are looking at actual sessions now. It’s probably something we don’t do enough of.”

One trap retailers want to avoid when prioritizing performance problems is relying strictly on customer reports of problems. By the time enough customers complain so the retailer can spot a trend, the problem may have become so full blown it impacts the credibility of the retailer’s brands in other sales channels.

“Customers aren’t always going to tell you why they abandon their shopping cart of if they were dissatisfied with the shopping experience,” stresses Crutchfield’s Weiskircher. “Besides, does any retailer really want to learn about a problem from a customer? If a customer brings a problem to the attention of the store manager, the manager might wonder what the staff is doing other than their job of servicing the customer. Customer feedback is important, but it does not give the whole picture.”

For retailers intent on including customer feedback when prioritizing web site metrics, a proactive approach is considered the best. Asking the customer to fill out a performance survey at the conclusion of a session via e-mail is one strategy. “This can help retailers understand where their site is weak and start to prioritize their investment in site performance,” says Bonny Brown, director of research for Keynote.

Some site performance experts counter that most retailers already know where their sites are likely to fall down, as they know the amount of resources put into building the site. “Retailers know what part of their sites were built haphazardly, and if they don’t, they are going to be in for some big surprises,” contends Nielsen/NetRatings’ Dougherty. “Retailers who rely too much on customer feedback to spot a problem end up following the problem, not being out in front of it.”

Still, customer feedback ought not to be completely dismissed. “Customer feedback can provide deeper insight to the impact of a performance problem,” says Brown, who acknowledges there is much debate over the weight to be given to customer feedback regarding performance.

Regardless of the data used to measure site performance, one aspect retailers and site performance experts agree upon is that e-commerce has matured to the point where retailers must adequately budget for performance monitoring and set clear performance goals across the enterprise. Scrimping on performance measuring is only going to create an environment that allows small problems to go unchecked and become full blown. Ultimately, such problems negatively impact sales and the retailer’s brand.

“The days of brainstorming about how to fix a problem are over,” says Brown. “The time has come for retailers to get more scientific about performance monitoring and have a clear idea of which metrics are most important to profitably running their business.” l

Peter Lucas is a Highland Park, Ill.-based freelance business writer. Picking up the pace of performance metrics

Once a retailer has aligned internal processing goals with performance metrics, the final piece of the puzzle is determining the frequency with which the measurement is taken.

The basic rule is the home page needs to load in 3 seconds, maximum. That’s down from 5 seconds three years ago, and 8 seconds in 2001, according to site performance experts. Most other key metrics ought to be measured every minute. The longest wait between measurements at the lowest end of the priority scale ought not to be more than 5 minutes.

Crutchfield Corp. measures site availability every 30 seconds and shopping cart and checkout every 1 to 5 minutes, depending on traffic. “Application response, site availability and page load time are three of our most important metrics that affect sales,” says Steve Weiskircher, senior director of information technology for Crutchfield. “We are very aggressive in measuring these.”

Retailers, however, have not always been as scientific. At the beginning of the decade, measuring frequencies and other benchmarks were set arbitrarily. That led to notably wrong assumptions. Despite the belief that the steady lowering of home page download speeds would significantly boost conversion rates, they only inched upward from 2% to 2.3% between 2001 and 2004 when the 3-second rule came into play, according to Geoff Galat, vice president of product strategy for TeaLeaf Technology Inc.

“Page speed is important, but its impact on conversion has not shown to be as great as some retailers think,” says Galat. “Retailers have to create performance benchmarks that allow them grow at the pace they want and that allow them to remain competitive against retailers offering similar merchandise. Benchmarks based on focus groups can still be pretty arbitrary.”

The best bet for retailers when setting realistic benchmarks is to weigh the level of service they want to provide against the resources they have available to make it a reality. “There is a cost to performance monitoring, but retailers can’t manage what they can’t measure,” says Bonny Brown, director of research for Keynote Systems Inc. “Retailers are better off leaving performance benchmarks to fact, rather than opinion.”