Over 80% of online ad effect is on offline sales

The study titled "When Less is More: Data and Power in Advertising Experiments" was authored by Garrett Johnson of the University of Rochester, Randall Lewis of Netflix, and David Reiley of Pandora, who were all conducting research for Yahoo! at the time of the study.

The Yahoo! researchers worked with an unnamed national apparel retailer to evaluate the effects of the retailer's advertising. They collaborated on a large-scale field experiment involving over 3 million Yahoo! users. For two weeks, Yahoo! users in the experiment's treatment group saw branded apparel ads from the retailer whereas users in the control group saw ads for Yahoo! Search. Relative to the baseline established by the control group, the experiment showed that the retailer's campaign increased sales by 3.6 percent or roughly three times the retailer's spending on ads.

Reiley said, "This apparel retailer approached us with an interesting problem: 'How do I know if my online ads work when 90 percent of my sales are offline?'"

The authors attacked this problem by matching customer records between the retailer and Yahoo!. Importantly, the authors combined customer-level online and offline sales data with a controlled experiment that allowed them to assess how much the consumers would have purchased in the absence of the ad campaign. They determined that 84 percent of the sales increase from the ads came from offline sales. Reiley added, "Without the experiment, the retailer could have erroneously concluded that the ads only increased online sales and not offline sales. Ironically, this could have led to underinvestment in online advertising."

The study notes that their novel experimental design can be valuable for companies seeking to measure advertising effectiveness. Lewis noted, "We'd run experiments with this retailer before, but this was the largest experiment where we used 'control ads' to determine which control-group members would have seen the ads. This allowed us to ignore statistical noise from the purchases of consumers who never saw the ads. We also discovered that we only needed to look at sales after the first ad because an ad can only affect you after you have seen it. These two tricks allowed us to improve the statistical precision of the estimated benefits from online advertising."

The improvement in precision from using control ads is critically important for managers making advertising decisions. Johnson explained, "Ad effectiveness estimates tend to be small, but also imprecise. Even with a study of 3 million users, standard methods to improve precision by controlling for customer's past behavior and demographics were less effective than most expected. However, by making full use of the control ads, we further improved by six times the statistical precision of our ad effects estimates. For managers, this improvement could be the deciding factor in learning whether online advertising has a clear and statistically significant positive impact. We hope the ideas in our design can help firms invest confidently in ad campaigns when they are likely to be profitable."