November 14, 2011, 10:14 AM adopts a new approach to site testing

At Internet Retailer’s design conference, the retailer will talk about better testing.

Lead Photo

Matt Butlein

When several years ago put an automatic product recommendation feature on its site, the online underwear retailer expected the tool to help increase conversions and sales. But that didn’t happen. Conversions decreased compared with the previous recommendation tool that enabled FreshPair merchandisers to manually create recommendation sets based on consumers’ site actions.

Multivariate testing, for which e-retailers run a series of tests on existing site elements simultaneously, also proved difficult to manage because, while those tests showed results, they didn’t answer the question of why consumers responded more positively to certain site elements than others, says president Matt Butlein. Butlein will discuss how these experiences helped to change the approach the e-retailer takes to testing and innovation at the Internet Retailer Web Design & Usability Conference 2012 in Orlando, FL, in February. Butlein will speak in a session entitled “Beyond best practices: Using innovation to take your site to the next level” Feb. 14 from 11:45 a.m. to 12:30 p.m.

The e-retailer now uses a thesis-driven split testing model to help prove the worth of new ideas for the site; Butlein says that under the model, the testing process starts with an idea, rather than a site element the retailer can test against. For example, in trying to find how could increase conversions, the retailer identified reasons why consumers will browse for bras online, but ultimately don’t make a purchase. It identified that bra sizing and fit were key issues. Having identified the issues, it came up with a variety of ideas that might help solve them, and then implemented and tested against them. Ideas come from all over the organization, such as from site analytics, survey results and customer service agents. The company also sources ideas from customers via post-purchase surveys and social media.

“This approach really permeates every aspect of what we do,” he says. “You know your category and your business better than any consultant who can recommend general best practices. It is hard from them to get at the fundamental business drivers that are unique to a company or category.”

Butlein says he hopes session attendees will learn how to approach site design challenges differently and understand that they can implement change. “Sometimes there seems like there are a lot of barriers—every organization has a sacred cow people think they can’t touch. But those things can change. More than anything I want attendees to feel optimistic that they can change any organization to be more innovation-focused,” Butlein says.

Internet Retailer’s editors asked Butlein to speak because of his experience in marketing and e-retail leadership at

comments powered by Disqus




From The IR Blog


Philip Masiello / E-Commerce

3 reasons retailers fall short in email and social marketing

Reason one: They’re constantly trying to sell their customer, rather than to help and engage ...


Rotem Gal / E-Commerce

7 surprising e-commerce trends for 2017

Consumers will engage with products and brands in new ways online in the year ahead.

Research Guides