No article on advertising is complete without quoting the great industry pioneer John Wanamaker. “Half the money I spend on advertising is wasted,” he famously said. “The trouble is I don’t know which half.”
In 2018, three economists attempted to figure out which half. Bradley Shapiro, Günter Hitsch and Anna Tuchman published a groundbreaking study on the impact of television advertising. The results are in: Wanamaker was an optimist.
In the largest study of its kind, Shapiro and his colleagues spent years combing through reams of data to find a correlation between advertising spending and the sales of 288 premium brands in 41,085 US supermarkets.
So what, you might ask? Television is over.
Many advertisers don’t agree. Worldwide, the biggest brands still spend more than $182bn a year on television commercials. Yes, that’s less than they now spend online, but it’s still a lot of money.
So what did Shapiro discover? First, that marketers waste extraordinary amounts of money on television commercials that do little to increase sales. Second, scientists publish mountains of research that doesn’t really tell us anything.
When I asked about the response to his study, Shapiro told me: “This research won’t make anybody happy.”
Can advertising actually increase sales?
It’s no secret that marketing people can be fond of convoluted words to sell their own work. (Think: “Transmedia inbound marketing to build buzz and show brand purpose.”) But the economics of advertising is pretty simple. Its job is to boost sales. If Heineken drops millions of dollars on television commercials only to end up selling a couple of hundred extra six packs … well, that’s a waste of money.
This is exactly what Shapiro’s study is about: how much more beer will Heineken sell if they advertise more? Not just in the short term (a couple of weeks) but also in the long run (up to twelve months later).
Economists call this “advertising elasticity of demand”. When you increase your advertising spending by 1%, how much do your sales increase?
To an intern working in Heineken’s marketing department, this might sound like a simple calculation: find out how much Heineken advertises, then calculate how much sales have increased. You’ve got your answer.
Unfortunately, it’s not that simple. Let’s say Heineken mostly advertises in places where people already drink a lot of Heineken, or they advertise during the Super Bowl when people are drinking a lot anyway. Let’s say they end up selling more beer. Who’s to say that advertising is the decisive factor? Higher sales could be caused by something else.
To really measure the impact of an advertisement, you need to differentiate between the “selection effect” (people who see your advertisement but already drink your beer anyway) and the “advertising effect” (people see your advertisement and buy your beer because of it).
In the pharmaceutical business, researchers have a tried and tested method for comparisons: they experiment. One random group of patients receives antibiotics, another group gets a placebo. A month later, you count which group has more throbbing boils.
So the question for brands and media buyers is how can you apply such an experiment to advertising? It’s not as though you can turn off all Budweiser advertising and turn on the Heineken commercials.
Or can you?
In a stroke of genius, Shapiro realised that such an experiment unintentionally occurs in America every day. American cable television offers different programming in different areas. So the commercials you see on television differ depending on where you live. Some parts of America are flooded with Heineken commercials, others only see Budweiser.
As a result, people who live relatively close to each other consume a completely different advertising diet.
Shapiro got to work. He built a database of all those people living in border areas and compared them against each other. Some were treated with the Heineken commercials (the medication), others got less Heineken (the placebo).
And guess what he found? Every euro Heineken spends on advertising yields less in sales. You read that correctly: they sell less.
Other beer brands do a lot better.
What’s going on here? Does it have anything to do with their 2018 commercial “Heineken, lighter is better”? (Chance the Rapper took to Twitter to call Heineken out for racism.) Or was it that commercial where a Jesus-like hipster figure holding a bottle of Heineken has a bunch of silly adventures on a cruise ship? It’s pretty funny, but it’s entirely possible people-who-like-Jesus hipsters are not the target audience.
Theories abound, but Shapiro has some bad news for people who like to speculate endlessly about “the science of advertising”: “What predicts success isn’t the brand or the commercial. It’s just a lot of statistical noise.”
Here’s the deal: the smaller the study, the louder the noise. Suppose I open a phonebook and call 1,000 people to ask if they ate pizza yesterday. Then I call another 1,000, and another 1,000, and another. I will have slightly different results for each of these little experiments. Sometimes I will find 50 pizza eaters, sometimes more, sometimes fewer. The bigger my sample, the closer I get to the True Pizza Density.
Advertising research is no different. When your sample is small – a brand hardly advertises or always spends the same amount, which means that there is no variation – your estimation of the effect won’t be very precise.
Statisticians can figure out the accuracy of such a measurement. They call this the “confidence interval”. Suppose you repeat the phone book poll an endless number of times, 95% of the results will contain the True Pizza Density within the confidence interval.
Now take another look at the beer graph, with line segments to represent the confidence interval:
As you can see, the confidence interval is very wide. This means that the effects of advertising seem great, but they haven’t been estimated precisely. There’s a good chance that the next time you repeat the Heineken experiment, you’ll get different results.
This problem is not limited to beer brands. It occurs in all supermarket brands Shapiro studied.
This is a very important conclusion. If it’s this difficult to judge whether or not an advertising campaign works, then it is impossible to find out why it works. The statistical problem gets even bigger when you want to compare different Heineken commercials. The sample is smaller, so you’ll get wider confidence intervals and no answers.
The effect of TV adverts on sales? Almost none
So what’s there to say about the effectiveness of TV commercials?
Mostly that you can’t expect too much from them. Yes, successful television ads exist. But for almost three-quarters of all campaigns, their effect on sales is statistically indistinguishable from zero. The median advertiser (that is, the middlemost advertiser in terms of effectiveness) saw a 0.014% increase in sales for 1% of extra advertising.
Shapiro estimates that, depending on your assumptions, 33% to 50% of brands make a profit from their television ads. And that’s an optimistic estimate. Shapiro only includes the cost of buying the commercial in his calculations, not the cost of filming it and the salaries of the marketing people.
Needless to say, this message won’t be popular at the marketing department’s next cocktail party.
Why are we only hearing about this now? Oddly enough, Shapiro’s disappointing results contradict heaps of television advertising research. Earlier meta-analyses – large studies that combine all existing advertising research – suggested that television advertising is seven times more effective than Shapiro’s study shows.
But again, Shapiro can explain: the disappointing results never made it to publication – so they haven’t been used in the meta-analyses.
Take a look at this graph, which only shows the positive results from Shapiro’s study. On the horizontal axis: the effect of commercials on sales. On the vertical axis: the accuracy and reliability of this research.
When you take the median value of these positive results, it does seem as though television advertising is seven times more effective than Shapiro claims.
But now, check out the graph that includes all Shapiro’s results:
We get a beautiful Christmas tree. And this is how it should be because when all the results are published, we see outliers on both sides. When the research is very precise, all the points will lump together around one point. When it’s less precise, they fan out to both sides.
If you’re missing half of the tree, this means you’re probably looking at “publication bias”. Non-results never see the light of day: academics don’t find them interesting, journals reject them, companies do not want them published. If you then summarise the studies that have been published, as researchers have done, you miss all the findings that were left somewhere to collect dust, which creates a distorted view of the true effectiveness of advertising campaigns.
Anything else to consider?
Of course, Shapiro’s study doesn’t account for everything.
Most importantly, these are premium brands with a lot of name recognition. Heineken doesn’t need television ads for most people to know its brand. But maybe it needed them back when Heineken was just an obscure Dutch beer brand.
Besides, Shapiro only measured the effect of local television commercials. Maybe national campaigns are much more effective – who knows? More to the point, this is the US! The country is swimming in commerce. One extra advertisement does not make a big difference. Would it be different in other countries, where people are less saturated with advertising?
So yes, there are criticisms to be made. But it seems like marketers mostly want to find ways to ignore the results. By now Shapiro has heard all the variations. Advertisers tell him they believe that the value of advertising is really in the long run, that an advertisement you saw when you were 15 will make you buy things when you’re 30. “We cannot falsify that,” Shapiro says dryly. Still, he is sceptical. When you cannot find an effect in the first 12 months, what makes you think you’ll find it in the 100th month?
Or advertisers think other companies are really bad at advertising, whereas their own is fantastic. As Shapiro says: “And if we show they are [bad at advertising], our methodology is wrong.” And finally, they retreat into general science denial: “They tell us that it’s so obvious that we are wrong that they are not even going to bother criticising our research.”
If you’re in advertising, you believe your advertising is profitable. Even if the research tells you it probably isn’t.
In another study, Shapiro looked into commercials for health insurance. A judge had ordered UnitedHealth Group, a large US health insurance provider, to cease all advertising for two years (there had been a little scandal regarding overly generous CEO bonuses). “It made no difference,” Shapiro tells me about the prolonged advertisement hiatus. “Not even a small shift in the market shares.”
And then what happened? Did they go back to advertising?
“Yep.”
“Why?”
“Excellent question.”
This piece first appeared on De Correspondent. It was translated from the Dutch by Katie Digan.
Dig deeper
The new dot com bubble is here: it’s called online advertising
In 2018, $273bn was spent on digital ads globally. We delve into the world of clicks, banners and keywords to find out if any of it is real. What do we really know about the effectiveness of digital advertising?