> Do you think Nike would be happy if their vendor just said, “yeah everything drives positive lift”.
It's fairly known in the industry (worked in a range of different ad-tech/marketing companies for around a decade), that more or less this is what makes Nike happy.
> They are employed by neutral third parties who make money regardless of the outcome of the study.
This is a bit naive. Same is true for home appraisers, funny how it always works out that the house appraised for just over what you offered (even in insane markets).
This issue is you might get paid this time for being honest, but you won't get paid again.
> completed by literal data scientists and statisticians.
I'm both of these. Saying that "adtech is bullshit" is absolutely too reductionist but you are wildly too naive about the reality of the industry.
Take A/B testing as an example. I've helped people run A/B tests for many years. I establish rigorous testing setups, make sure people understand confidence in the results, distribution of possible outcomes, etc. I'm honest in my work and so are the people running the tests.
But the truth is that A/B test on customer populations are not controlled experiments. One thing that literally no one in the industry does is go back and review the results of the past years A/B tests. The reason why? People don't really want to know. However, it's obvious these results can't be quite right because teams will run 10 A/B tests in a year, each claiming a 5% improvement in conversion, but clearly you don't see this cumulative impact of a >60% increase in conversion over time.
No one want to know this though, because knowing this helps nobody. Nobody is outright lying, but nobody wants to ask. The one time I worked outside of adtech but saw a problem where model results were almost entirely random noise I got in huge trouble for pointing this out.
Same is true across the board in the adtech world.
Someone else recommended "Subprime Attention Crisis by Tim Hwang" and I very strongly recommend that you read this. It's very well researched and was so inline with my experience that I found it so obvious as to almost be dull.
It's fairly known in the industry (worked in a range of different ad-tech/marketing companies for around a decade), that more or less this is what makes Nike happy.
> They are employed by neutral third parties who make money regardless of the outcome of the study.
This is a bit naive. Same is true for home appraisers, funny how it always works out that the house appraised for just over what you offered (even in insane markets).
This issue is you might get paid this time for being honest, but you won't get paid again.
> completed by literal data scientists and statisticians.
I'm both of these. Saying that "adtech is bullshit" is absolutely too reductionist but you are wildly too naive about the reality of the industry.
Take A/B testing as an example. I've helped people run A/B tests for many years. I establish rigorous testing setups, make sure people understand confidence in the results, distribution of possible outcomes, etc. I'm honest in my work and so are the people running the tests.
But the truth is that A/B test on customer populations are not controlled experiments. One thing that literally no one in the industry does is go back and review the results of the past years A/B tests. The reason why? People don't really want to know. However, it's obvious these results can't be quite right because teams will run 10 A/B tests in a year, each claiming a 5% improvement in conversion, but clearly you don't see this cumulative impact of a >60% increase in conversion over time.
No one want to know this though, because knowing this helps nobody. Nobody is outright lying, but nobody wants to ask. The one time I worked outside of adtech but saw a problem where model results were almost entirely random noise I got in huge trouble for pointing this out.
Same is true across the board in the adtech world.
Someone else recommended "Subprime Attention Crisis by Tim Hwang" and I very strongly recommend that you read this. It's very well researched and was so inline with my experience that I found it so obvious as to almost be dull.