Testing, testing 1, 2…always

Retailers can (and should) constantly test site modifications, say IRWD speakers.

Katie Evans

E-retailers have long heard the much touted benefits of A/B and multivariate testing, but too many do it haphazardly and without a clear definition of how they will measure success, Dan Siroker, founder for Optimizely, a web testing services firm, and Joan King, e-commerce director, site management for home goods retailer Crate & Barrel, said today in a session entitled “Lessons learned from 75,000 A/B and multivariate tests” at the Internet Retailer Web Design and Usability Conference 2013 in Orlando. In a discussion with ample audience participation, the duo offered tips to run tests better and also shared what they’ve learned about consumers in the tests they’ve administered over the years.

Siroker, who in the past has worked for Google Inc. and also for President Barack Obama’s campaign web site, says retailers should first define success. “Too many companies start moving around levers without developing a clear understanding of what they are trying to optimize,” Siroker told the audience. “For e-commerce, quantifiable success metrics could be increased average order value and revenue per visitor.”

Siroker culled several interesting examples of how he used testing during his time working on President Obama’s campaign web site. For example, digging into his analytics, Siroker found that, of those who visited President Obama’s web site while he was running for office, few signed up for e-mail. However, those who did sign up to receive e-mail were much more likely to donate money. So his team tested a handful of different calls to action to register to receive e-mail. The winning prompt boosted the percentage of site visitors who signed up for messages from 8.3% to 11.6%, resulting in 2.9 million new e-mail addresses and an additional $57 million in contributions.

King of Crate & Barrel says her company watches small groups of eight to 10 consumers interact with its current site and also mock sites featuring potential tweaks using programs such as WebEx or GoTo Meeting. It then tests best performers using A/B or multivariate tests.

For example, Crate & Barrel is testing how to display such text as product details and product overviews. It saw from test groups that consumers were overwhelmed by having to scroll down the page and scan large amounts of text to find something. So the retailer is testing placing information in what King calls drawers that consumers can click to open and expand. In one version of the site, the product overview drawer is automatically displayed and visitors can click to access other text such as the product detail information. In another, the product detail drawer is open and users can click to expand other fields. And in a third, all drawers are closed and a user must click to expand each one.

Another tip the two experts shared is that less is often more. The speakers pointed to an example from a flash sale housewares retailer In its sign-up screen, it eliminated two fields—one asking visitors to re-enter their e-mail address and another requesting the shopper’s ZIP code. Eliminating the two fields prompted an 8.4% increase in e-mail sign ups.

King provided another example where removing navigation tabs during checkout for its sister brand CB2.com increased conversion rates by 15%. Offering too many options, such as the option to abandon the purchase process, ultimately hurt conversions, the retailer discovered. 

Another fact the two professionals learned from testing over the years is that every word counts. “If you want someone to do something, ask them,” Siroker said. He pointed to an example of a site that wanted visitors to sign up for a free trial and changed wording on the button to start the sign-up process from “Free Trial” to “Try it Free.” The site increased sign-ups by 15%.

Crate & Barrel had a similar experience. Changing the phrase “Choose items” to a more direct “Add to Cart” resulted in 4.2% more shoppers adding the products to their carts. In another example, Siroker described how a simple change from the word “Submit” to “Support Haiti” on a page where visitors could donate to help victims of the Haiti earthquakes increased the amount donated by around 15% on average per visitor.

Both speakers stressed the importance of realizing when a potential change is not a smart move. “Fail fast,” Siroker said. He used an example of a gaming web site that wanted more visitors to view its videos and so it moved the Videos tab at the top of the screen from its original spot of nearly last in a long of tabs to the front of the line, hoping that the more prominent display would increase views. Clicks on the Video tab fell a whopping 92.3%—exactly the opposite of what the site was hoping for—likely because the site attracted many repeat visitors who were accustomed to seeing the tab in the same spot and did not put in the effort to search out videos when they could not find them in their familiar home.

King described a similar instance when Crate & Barrel, assuming shoppers would want to know the amount of taxes as early as possible in the checkout process, began telling customers the total price of their order in an earlier checkout screen. The move led to a drop in consumers finishing their orders. In another experiment, Crate & Barrel added average star ratings to what it calls the spill page, or the page that shows all versions of one type of product, such as espresso makers. Extending the average star rating from just the product page to the spill page as well resulted in a 10% drop in conversions.

“We did it because everyone seems to love reviews,” King said. “But we found it’s better to test then to just go out and implement.”


Crate & Barrel, Dan Siroker, e-commerce, e-mail marketing, Google, IRWD 2013, Joan King, merchandising and design, Obama, One Kings Lane, Optimizely, Performance management, site testing