Technology that processes large volumes of data fast becomes more affordable—and more crucial—for e-retailers' success.
Members-only retailer BeachMint Inc. stores data about every consumer action on its e-commerce site—clicks, views, transactions, searches, shares, the type of device used to browse—dating back to its first brand launch two years ago, says Doug Cohen, director of business analytics at BeachMint. "If you go to the site and linger over a product and don't click, we store and know that," he says.
That's in addition to data BeachMint keeps about its customers' social media activity and the roughly 13 data points per member that come from a style quiz shoppers complete during sign-up, he says. And BeachMint has millions of customers, Cohen says, though he declines to give a more specific number.
Keeping those activity logs isn't just a compulsion. BeachMint has mined that data to figure out better ways to recommend products, lifting conversions by more than 10%; to personalize e-mails, more than doubling conversions; and to solve other tricky business problems, such as how to choose which items to buy for the next season—which BeachMint decides based on feedback from pictures posted to test groups of Facebook users, Cohen says.
If BeachMint had launched just a few years earlier, however, the hardware to store all that data—it rents server space from Amazon.com Inc., accessing its data over the Internet—and the technology to query and process it effectively—the e-retailer uses Hewlett Packard's Vertica software for managing its database and Pentaho Corp.'s software for analytics and reporting—would have been neither mature nor cheap enough to use, he says. BeachMint instead would have been limited to older technologies built to answer questions common to many businesses, such how many clicks and conversions a marketing campaign produced.
Today, it's realistic for e-retailers like Beachmint to parse the trillions of data footprints consumers leave as they traverse the web to target them with highly personalized offers. They're taking advantage of the boom in so-called "big data" tools that offer more storage capacity, greater computing power and more rigorous, flexible ways of analyzing retailers' data than ever before. These tools aren't cheap—implementations still typically run in the tens of thousands of dollars—but they no longer require the kind of multimillion-dollar supercomputers that would be out of the reach of all but the biggest retailers.
Prices are coming down as the demand for big data technology and services grows. This market will grow at a rate of 31.7% per year from 2012 to 2016, predicts technology research firm International Data Corp., seven times the rate of the growth for information and communication technology overall.
With that market growth comes more readily available, reliable and mature technologies at lower costs, enabling retailers of all sizes to use big data storage and analytics. Their goals are to learn about shoppers in as much detail as possible, including their transactions, demographics, preferences, clicks, views, tweets and the like, in order to inform all types of decisions, from marketing to merchandising, and in a timely manner. Some are finding those goals are not only achievable, but that big data changes how their business operates.
What is big data? Big data itself has a number of characteristics that set it apart from the information yesteryear's businesses may have stored and analyzed. First, as the name suggests, it often involves datasets that are terabytes or larger in size. (A byte is the amount of data needed to encode one letter of the alphabet; a terabyte is equal to about 1.099 trillion bytes, or the amount of data on roughly 1,500 standard CD-ROMs.) Previously, common analytics technologies used by retailers reached their maximum capacity at about one-tenth of a terabyte.
Secondly, big data comes from many sources, including outside of a retailer's e-commerce site, call center or stores. It might take many forms, from a short Twitter post to a video uploaded to YouTube. That means a retailer is likely to have certain types of data about some customers, and very different data for others. Mining that data is often not possible with older analytics tools built to handle complete record sets in standard formats, such as the name and address of a customer, what she bought and when.
Finally, big data often changes quickly, which means running a new query usually requires loading a fresh data set to capture recent updates, whereas older analytics tools could generally get by using data compiled over the past days or months, says David Floyer, co-founder and chief technology officer of Wikibon, an online community for technology professionals.
Even if older technologies could capture and query all the available data, they often do so too inefficiently to meet a retailer's needs. "The inabilities of the past were more a function of computing speed—the answers with large datasets would not come back fast enough to be useful," says Cohen of Beachmint.
BeachMint learned firsthand the value of big data technology when an intern, a post-doctoral student at the California Institute of Technology, designed a big data recommendation engine and tested it against the retailer's older, less data-heavy engine.
In the past, Beachmint recommendations were based on a customer's browsing and buying history and product attributes—as in, if you purchased or viewed a button-down shirt you might like another shirt in a similar style. But for BeachMint, its catalog of thousands of products offers far less information than the store of profile information it has on its members. The intern wrote a program to compare an individual's profile with that of other members and suggest to her products that customers deemed similar to her liked.
The profile-based recommendations convert up to 20% more often than the product-based ones, Cohen says. They also allow BeachMint to show a new member extremely relevant products as soon as she fills out her profile, before she's browsed or bought anything from the retailer.