The point of the article is that the sale would happen regardless of the ad.
Ie showing an ad on Google to someone who types 'Ebay' in the search string is a waste of money - since Ebay shows up as the first natural result anyways.
However, this leads to the other flaw in the argument, where eBay has been the company most famous for abusing Google's ads, and using them in a completely nonsensical way.
I'm sure you've noticed that a search for any noun called forth an ad for eBay. You're enticed to "buy Dog Vomit on eBay" when your dog is sick. And then, even if it was something eBay plausibly sold, when you clicked through it'd take you to a generic eBay search page or, often, a landing page that was not useful at all.
It is very unfortunate that the extremely interesting ebay study is being ignored by the SEM community because ebay has some bad adverts.
The methodology and findings should be something that advertisers look at and learn from (i.e. that traditional methods of tracking do not always do a good job of showing value) but instead people just like to point fingers and call names
It is not ignored, because eBay has some bad adverts. It did attract significant attention more than a year ago (the study was conducted in 2012, and media reported on it in early 2013).
"eBay has no idea if online ads work" would be a better title. Or "eBay can't get their online ads to work".
If you looked at eBay's search advertisements in the last years you probably have seen very ridiculous ones, like:
Love.
Buy it cheap on eBay
Low Prices. New and used.
This is a significant waste of money, and seems endemic at eBay, not a rare occasion. Though it is too easy to point at this and discard the entire study, it is a relevant prior.
As a control they take MSDN/Yahoo vs. Google. The control should be a different advertiser, not a different advertisement network.
In the paper they mention removing the term "eBay Shoes" from their keywords, because they rank organically for that. They should have never added it, because it is a bad keyword for them. They should advertise on the brand name shoes they are selling. This ties back to the bad adverts by eBay: Fixing your mistakes and seeing change, is different from performing a controlled experiments on a healthy advertiser.
They muse about "the types of consumer that may already be informed about the advertiser’s product". This adds a few problems: those customers may have been informed through the very advertisements they now claim don't work. Suddenly shutting down advertisements does not negate the effects of running advertisements years before that. The paper does not investigate the effect that ads have on brand recognition and brand trust. Usability tests show people clicking on advertisements first, because then they expect quality and a simple commercially interesting proposition.
I do think the problem they are describing "some customers may be so aware of a brand, that advertising to them becomes worthless" is valid. Maybe ad networks could help here and implement a form of "demarketing" next to their "remarketing" options: Once a user has enough knowledge of the brand, stop advertising. But a better (and currently available) method could be to apply remarketing and give these power users tailored advertisements. An example could be a buyer's program, promoting a new product from a well-purchased brand, or a referral bonus. To continue offering them advertisements like "eBay shoes" does not make much sense.
I am aware of the significant attention this paper got about a year ago. In my filter bubble (and I am in a massive filter bubble with regards to this kind of thing) the attention was similar to the rest of your comment - focusing on ebays failures of execution rather than the failure of measurement.
Other advertisers do not make the same mistakes as ebay with regards to landing pages, adverts or keywords selection but many make the same mistakes when it comes to measuring their advertising value.
Now, perhaps the errors caused by faulty measurement are not as important for other advertisers and I can see why this might be the case (for the reasons you list about how ebay is running their PPC) but what should be an opportunity for the SEM industry to look at how their work in valued and measured has instead turned into something else
> As a control they take MSDN/Yahoo vs. Google
They also used a geographic split as a control. E.g. if in the past areas A and B have performed similarly then pick one of them at random to stop showing ads in.
AdWords geo-targeting combined with this technique is a way for other advertisers - regardless of scale - to run similar tests. But we won't see many doing this because of the attitude that it is ebays execution that sucks rather than their measurement
The paper sure is interesting and should add value to the debate and approaches to SEM. To discard it by scoffing at eBay's execution would indeed be your own loss.
My biggest take-away is the saturation of advertisements and the effects it can have on running ad campaigns. Yours seem to focus on the common correlation-causation misinterpretation, and faulty measurement approaches.
I do think there are enough problems with the paper to not grant it the authority to speak for an entire industry (as Slate put it). eBay is just one (and a very big and unique) advertiser in this arena, and every move it makes (even to measure it) changes the entire ad network.
> They also used a geographic split as a control.
Thank you for this correction.
Aside: The problem with eBay's non-profitable advertisements seems to me a beautiful problem: The paper authors subscribe to the benefits of saved costs. I do not think that the marketing department of eBay subscribes to this benefit. For them, and external ad agencies, the more ads your run, the more clicks you get, the more you get rewarded, the better your monthly reports look. People at eBay's online marketing department must be smart enough to realize that many of these ads were worthless to the company, yet it was worth it for them to empty the monthly budget on a list of generic keywords.
People at eBay's online marketing department must be smart enough to realize that many of these ads were worthless to the company, yet it was worth it for them to empty the monthly budget on a list of generic keywords.
I read somewhere that alleged cookie-stuffers claimed that eBay's staff knew about the cookie-stuffing all along, and encouraged it so long as it increased the sales attributed to their own work in courting affiliates.
If true, these are both cases where incentive design (and measurement) are as important as people's skill with their craft.
I now agree with most of what you say - as you point out we were just looking at different parts of the same thing.
Your aside comment is really interesting. I'm not sure what the solution here is - more accurate systems of measurement lose the tight feedback loop that make online ads so successful, but simpler measurement is wrong. Some sort of combination of the two is the answer, but how that exactly works in practice I have no idea!
Well, there's nothing to say that eBay's research lab was using these horrible PPC ads in their tests. In fact, I would guess that they had to hold the landing page as a constant (e.g. homepage when "ebay" was the keyword), or otherwise account for that variable in the study design.
For instance, the study notes that, for one test, they simply turned off their PPC campaigns for one geo and compared results to others. So, presumably, even if they were using their horrible PPC ads during this test, they would have been equally horrible across geos.
It's worth noting that display advertisers can and do account for this by running a control group that sees public service announcements instead of the ads they'd normally run.
Ie showing an ad on Google to someone who types 'Ebay' in the search string is a waste of money - since Ebay shows up as the first natural result anyways.