November 14, 2011, 10:14 AM adopts a new approach to site testing

At Internet Retailer’s design conference, the retailer will talk about better testing.

Lead Photo

Matt Butlein

When several years ago put an automatic product recommendation feature on its site, the online underwear retailer expected the tool to help increase conversions and sales. But that didn’t happen. Conversions decreased compared with the previous recommendation tool that enabled FreshPair merchandisers to manually create recommendation sets based on consumers’ site actions.

Multivariate testing, for which e-retailers run a series of tests on existing site elements simultaneously, also proved difficult to manage because, while those tests showed results, they didn’t answer the question of why consumers responded more positively to certain site elements than others, says president Matt Butlein. Butlein will discuss how these experiences helped to change the approach the e-retailer takes to testing and innovation at the Internet Retailer Web Design & Usability Conference 2012 in Orlando, FL, in February. Butlein will speak in a session entitled “Beyond best practices: Using innovation to take your site to the next level” Feb. 14 from 11:45 a.m. to 12:30 p.m.

The e-retailer now uses a thesis-driven split testing model to help prove the worth of new ideas for the site; Butlein says that under the model, the testing process starts with an idea, rather than a site element the retailer can test against. For example, in trying to find how could increase conversions, the retailer identified reasons why consumers will browse for bras online, but ultimately don’t make a purchase. It identified that bra sizing and fit were key issues. Having identified the issues, it came up with a variety of ideas that might help solve them, and then implemented and tested against them. Ideas come from all over the organization, such as from site analytics, survey results and customer service agents. The company also sources ideas from customers via post-purchase surveys and social media.

“This approach really permeates every aspect of what we do,” he says. “You know your category and your business better than any consultant who can recommend general best practices. It is hard from them to get at the fundamental business drivers that are unique to a company or category.”

Butlein says he hopes session attendees will learn how to approach site design challenges differently and understand that they can implement change. “Sometimes there seems like there are a lot of barriers—every organization has a sacred cow people think they can’t touch. But those things can change. More than anything I want attendees to feel optimistic that they can change any organization to be more innovation-focused,” Butlein says.

Internet Retailer’s editors asked Butlein to speak because of his experience in marketing and e-retail leadership at

comments powered by Disqus




From IR Blogs


Don Davis / E-Commerce

Is China still an online opportunity for Western brands?

On the ground, China’s economy looks quite strong. And Chinese consumers crave Western goods.


Barry Shteiman / E-Commerce

Three things all retailers need to know in the data breach age

Hackers often use stolen credentials to break into networks. There are tools that can spot ...