Contrary to the belief that advertising is less data-driven, the complexity and dependency on feature-rich data sets has increased over time. Redundant targeting sometimes happens due to the ad objective of maximizing expected revenue over cost. However, it's more nuanced in practice.
Advertisers aim to optimize expected revenue over cost within a 'payback period', which is typically 1-3 years for large advertisers. This is calculated through customer acquisition cost, retention, and incrementality (the probability of an ad causing conversion).
Only advertisers themselves can effectively calculate incrementality due to access to their specific conversion/retention metrics. This incrementality, along with the optimal mix of channels and ad spend, is the ongoing challenge for sophisticated advertisers. It involves multi-objective optimization across millions of ad assets, campaigns, and targeting criteria.
Privacy regulations since 2015 and subsequent laws like GDPR and CCPA have led to more reliance on probabilistic modeling for targeting. The entire pipeline of targeting, engagement, conversion, and retention forecasts are now based on probabilistic models.
While ad networks offer simplified scaling solutions like 'target roas' and 'campaign budget optimization', they're more useful for average advertisers with limited internal resources - eg, they can't justify hiring ML SWEs, quant traders, technical PMs, etc.
Advertising has become even more data-driven and arbitrage gains for sophistication have increased. Profound gains can be made with investments in marketing and forecasting science, similar to the operations of a quant trading firm.
Source: I've managed $10B+ through automated ad spend systems since 2012.
How much more effective would ad spend be if ads were solely served to people with a decent bit of disposable income? I would guess the return on investment is almost all due to buyers with cash to spend more than anything, and those 40% of Americans with like $400 in the bank (if those popular stats are to be believed) are merely an expense serving them advertising content for products they are unlikely to ever buy.
You already see this with some goods so its certainly at least somewhat effective, e.g. luxury watch ads in beverly hills contrasted with military recruitment billboards in south LA.
"For users who opt in, Brave can deliver better quality ads, without the risk of personal data leakage. We do this by using local machine-learning to understand the user better, and making local decisions as to which ads should or should not be shown, and when (the user controls all of this). Furthermore, the user gets 70% of the ad revenue for browser-private ads."
Sorry. This is just wrong. You don't nor never will have the volume of data that the major ad platforms have collected for 15+ years. You don't have the pure analytical power of these companies in compute capacity. You will not serve better ads by disabling third party retargeting platforms.
There are many positive ways to spin an obvious blockchain-craze inspired endeavor, and I applaud the Brave team for blazing a trail, but you are calling out all the wrong value propositions.
Fraud is not as big of a deal as you state it is. It is not a problem for the primary ad networks to which the lions share of the ad revenues goes to in the first place. Companies like WhiteOps and Moat already do a great job of fraud detection when an advertiser is not buying from one of the popular and safer exchanges.
I applaud your conviction, but the problem with advertising is not solved by attempting to de-throne Google and Facebook.
This sounds like a good fit for a friend of mine, but he'd need some kind of documentation to even consider showing up. Do you have anything about the platform that you're willing to share?
Per a SWAT officer - his team has not been activated, which means the situation is under control. Apparently an attempted murder/suicide. Female shooter. 10-20 shots fired and multiple people being treated for gunshot wounds. No news as to what happened to the shooter.
I'd suggest everyone reading exercise extreme caution with unverified information like this. There are already widespread conflicting reports of female vs. male, wearing body armor vs. not, etc. etc.
Don't doubt for a second that 4Chan and the like are eagerly spreading disinformation as we speak. You don't know the source of this information. Wait for official information before forwarding anything.
Honestly, it's times like this that make me incredibly fucking sad for the future of humanity. It's bad enough that there are people dealing with a tragic and imminent situation, but to then immediately make this about yourself is absolutely loathsome.
Also, the comments on the live YouTube broadcast of 'they deserved it' or 'serves them right' or 'what now, NRA' are fucking repulsive. Seriously, I don't know how to fix this problem, but it makes me so incredibly angry to realize just how few people can empathize with the plight of others.
I think that under the circumstances links/citations are required. False rumours are rampant and it's too easy to contribute to their spread (even unwittingly).
> Do you expect the SWAT agent to stop and write a blog post so this person can cite it?!
The poster (most likely) did not talk directly to the SWAT agent. I was asking for a link to the poster's source containing the SWAT agent's account.
> Unless you have reason to suspect this three-year-old account is suddenly spreading harmful lies, this is as credible as anything else we're hearing.
I don't doubt the poster's good intent, but well-meaning people are just as effective a conduit for the spread of disinformation as the malicious (indeed, they can be even more effective).
This three-year-old account might believe this to be absolutely true, because they were told it by someone they trust. And that person was told it by someone they trust. And so on, through dozens of people if not more, and all it takes is one person in the chain to start exaggerating/fabricating information and everyone is forwarding false information.
> this is as credible as anything else we're hearing.
You're right there. And that's the point: nothing we are hearing is credible. Treat it as such.
To be clear, this is an extreme circumstance where people hunting for an explanation could easily gravitate toward misinformation is a uniquely destructive way. So, sure, I don't have a reason to suspect OP is a liar; I have a reason to suspect that all information right now is likely untrue unless confirmed otherwise.
How does it affect anyone's lives if this is a rumor or true? The situation will unfold exactly the same whether it is live updated on HN or not. I'm inclined to believe that OP at least is acting in good faith, because their username looks like a real name, they claim to be an engineer at Netflix, and have a long unrelated post history. Maybe they are honest, but got bad information, or maybe it is true information. But still, who cares? Getting information 1 hour earlier will only satisfy some gossip desire and not actually affect your lives.
Every comment ever posted on HN is an excuse for other users to argue about epistemology. Even your question is likely to become an argument about how we can be certain of the net harm caused by misinformation after shootings, with citations about that. HN culture is terrified of false beliefs even when it doesn't matter, so it's no surprise most of the comments are about the fear of false beliefs when it might matter.
San Bruno police just held a mini press briefing, indicated that their is one person dead with a gunshot wound that seems to be self-inflicted that would appear to be the shooter; but certainly was a lot less willing to commit to that as a fact than you (or some other news sources, even before the briefing) are.
Reporting on the specific activities of police during a shooter event is generally considered bad form. Please don’t share information about where and/or how police are/aren’t deployed in realtime.
As someone who is a product manager in the commercialized AI SaaS space, the most important pieces of feedback I would give a new PM here:
1)Don't let your -brilliant- colleagues try to force their -brilliantly complex- solution of a problem - clearly define market problems, and don't let the team try to go the route of trying to force fit a solution to a market problem. Market problems come first.
2)Frame the market problems appropriately for your ML/AI teams, and practice trying to frame the problem from a variety of angles. Framing from different angles promotes the 'Ah-ha' moment in terms of the right way to solve the problem from the ML side.
3)Don't commit serious time to a model before having a naive solution to benchmark against. Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
>Always have a naive solution to compare against the AI solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed bandit/Thompson sampling.
This cannot be stressed enough, optimism bias will always push the scientist towards the 'more interesting/complete/new' method and model, but a seasoned practitioner will have the discipline to always establish a baseline (<1 days work).
In my experience, simplifying a solution over time is the best way to ensure its adoption and long-term success. That applies both to UX/UI and the supporting computation/ML.
They can work in conjunction - In the domain of website optimization, where visitor attributes are often greater predictors of value than website content, a system driven by search space optimization can more easily take into account changes in those variables - eg; time of day, traffic source, device type - and incorporate those inputs to climb multiple 'hills' simultaneously.
The allocation of traffic based on the evolving optimal search space (blue button for visitors from Facebook) can then be driven through an MAB or something similar.
As someone who has been responsible for a bit north of $1B in profitable, attributable digital ad spend in my career, I can say with conviction that the problem comes down to analytics and misalignment of incentives, NOT the performance of digital media channels.
1)Agencies charge on a % of total media spend, and are thus incentivized to spend more.
2)Advertisers net benefit from expected lifetime value and revenue generation, but are often reticent to share this type of information to an agency.
Agencies are commonly unable to get access to business-level health metrics such as churn, RPU, LTV, and thus optimize to top of the funnel metrics that often do not correlate with attributable lift but do correlate with showing the value of increased levels of spend. Such metrics include click-through rate, viewability, brand awareness and safety, fraud mitigation, etc.
This, combined with the improved ease of use for major digital platforms (I know quite a few startup CEOs who manage all of their PPC/Facebook ad spend), is why the agency model has started to fail. And that is a good thing.
The less intermediaries that touch an advertising campaign, the less likely it is that we as consumers will see an irrelevant ad.
It is, better to say, a sunset for the industry as such.
Digital ads had the most wow effect at the time of google's text ads and Internet yet not being a general demographics' thing.
Back then, they were able to show numbers, but not now.
First generation internet users were mostly highly technically literate, high income professionals. Now, the user ratio has reversed.
You can argue that putting efforts to find a needle in a haystack was still paying off when the majority of needles were of incomparably higher marginal value than they are now: consumer goods clicks, say, are lower value than commercial equipment sales clicks, yet you still have to put an equivalent or greater effort to datamine somebody to make them buy an accursed face lotion even if the number of face lotion buyers is 1000 times bigger.
This is a tragedy of the Internet ad industry. There is a finite amount of eyeballs, and the amount of companies with a substantial money wanting to sell you a face lotion will always eclipse the amount of companies who market unique, relevant, specialty products one may actually be actively looking to buy.
>This is a tragedy of the Internet ad industry. There is a finite amount of eyeballs, and the amount of companies with a substantial money wanting to sell you a face lotion will always eclipse the amount of companies who market unique, relevant, specialty products one may actually be actively looking to buy.
It has little to do with having a large amount of money to spend on ads and more the level of severity of the problem your product solves for the 'eyeball.'
Humans have some needs that are greater than others, and ad dollars spent at alleviating pain or providing for those needs will always get at the most eyeballs. Entire industries - born online - have developed around capitalizing on solving for these needs.
I need money - Predatory loans and credit card offers.
I need to be skinnier - Nutraceuticals (Dr. Oz), weight loss, vitamin crazes.
I need to be beautiful - Skin care, facial lotions.
I need to be less lonely - Online dating, pornography.
The list goes on. The point is - there are more people looking to be skinny, beautiful, rich, and married with children than there are people looking for 'commercial equipment sales.'For now, and until targeting gets smarter and consumers opt to share more data about themselves, those products will win the majority of ad clicks.
We'll get there. (eye roll) Maybe decentralization is the way? :). I'd love to give the Google/FB duopoly a kick in their 'walled garden.'
>unique, relevant, specialty products one may actually be actively looking to buy
This is because the number of people who buy normal lotion is 1000x the number of people who buy unique specialty lotion. Why would the advertising be different than the market?
>First generation internet users were mostly highly technically literate, high income professionals. Now, the user ratio has reversed.
The internet reached the masses - how is that a negative for advertisers? They have more people to advertise to.
>yet you still have to put an equivalent or greater effort to datamine
This isn't true. For lotion you buy lotion related keywords, for commercial equipment you need in-depth research about those products, use cases, industry terms, and the queries people use.
>This is because the number of people who buy normal lotion is 1000x the number of people who buy unique specialty lotion. Why would the advertising be different than the market?
No, I can't imagine anybody randomly buying a random face lotion online, and I say that as somebody who did a stint in adtech for a few years with access to "crown jewels" of a major advertising brokerage conglomerate. People barely click on FMGC ads even if being force fed, I would've shown you the digits if not for NDA. Their purpose is impressions, even if they are not explicitly billed on PPM basis.
The lion share of ad inventory of any tier 1 ad vendor are for "stuff people buy in Walmarts," and yes the total revenue from FMCG ads is mind boggling and defies any attempt at rational understanding.
Even if the sole point of an ad is to keep the product on top of somebody's mind, it will be of very very little payoff to an advertiser. Though, one can reason that a gigantic big fat FMCG co. bathing in cash can still easily afford doing so even if is patently stupid. This is because even if their bang for buck is approaching zero, it is still better than none for them.
>This isn't true. For lotion you buy lotion related keywords, for commercial equipment you need in-depth research about those products, use cases, industry terms, and the queries people use.
I counter your argument. In both cases, the adtech efforts needed to sustain a barely functioning campaign eclipse all other hurdles. Extreme targeting is the key - even if it sucks, it sucks less than investment in almost anything else including £1k per hour big name marketing consultants.
How it looks on technical side: huge effort at industrial scale purchasing of "cubes" - huge databases of cookies with their statistical and fuzzy logic data from thousands of companies, including the black hat scene. Other than cookies, analogous data come from email list vendors, IMEI/phone number db vendors, vendors of stolen contact lists, search histories, GPS data and etc. All of this is barely enough to for a sustained campaign for F500 FMCG.
putting efforts to find a needle in a haystack was still paying off when the majority of needles were of incomparably higher marginal value than they are now
Was this back when Google included the referrer URL with each needle?
As a former fellow digital media manager who worked for a WPP agency - you're right.
I would just add that, agencies have clients that are nice to have in their portfolio but they give more work than they invest.
So you have a business model built on % of investment - something that worked very well for massive TV Campaigns budgets (plus the rappel you'd get from TV Networks) - applied to time consuming campaigns that required a lot of FTAs.
I just don't agree much on the "ease of use" - the digital media landscape is quite complex.
A proper digital media campaign (part of a media campaign) has media managers, digital media managers, PPC Managers (let's say they manage FB and Google advertising for the sake of simplicity), AdOps, and Programmatic Teams - you can add on top of this SEO teams, Social Media teams, etc.
Yet advertisers don't want to move one inch on this business model - let me tell you i've seen them criticizing how many hours a proper Search campaign takes, with no knowledge of the matter. Some, like you've said, are bringing in people for digital media management, campaign management ... hell some are building their own PBUs!
I don't think it will work. It's not something new: it was done in the past with TV, and they come to realize the cost of the know how being stuck on their end it's too expensive.
Oh and P&G, and many other advertisers are to blame here as well - the marketing/brand managers don't want cuts on their budgets, so they make reckless decisions, wasting money just to claim they invested it and will require more budget in the next fiscal year.
Media owners are also suspects, the greed made them dependent on Google and Facebook... now their brands are diluted.
Unfortunately agencies failed with the golden egg goose - attribution modeling... attempts were made but it's not something trivial.
the less likely it is that we as consumers will see an irrelevant ad
As a consumer I don't want to see any ads, in particular I don't want to see well targeted ads because they are the ones most likely to change my behavior.
Why would you see such change in behaviour as something bad? Personally, I'm thankful for targeted ads from companies that I love about new products that I don't follow myself. What's your experience with them that is so negative?
If you only had perfectly targeted advertisements, you would continuously be pushed to spend more money and attention to products and services. You might say 'I am in control', but at the minimum it wears you down to stay disciplined. Since time and mental clarity are rather high on my list of important things to protect, I view all push-advertisement is a problem. But I prefer badly targeted ones.
It's different when you want to research something, like buying a mobile phone, when you go to their channels and pull in reviews, advertisements, etc.
You're all pushed towards something all the time by everyone around you. Trying to be in complete control feels weird, to be honest — are you sure it's worth it? Yes, I know that I am sometimes convinced to buy something I wouldn't think of buying otherwise, but the outcome is usually not half bad and I don't understand why I should guard myself from it.
I think advertisement should be taught at school, so everybody can recognize when they are being pushed/nudged into feeling something. Because I think it does a whole lot of damage to society (obesity, stress, debt, etc).
I know it sounds depressing like that, but have a nice day :)
Would you like Netflix to show you trailers before you could watch a movie?
Come to think of it, they do. They show you trailers at the top of the page. I'd rather they just had a 'show me trailers' button, I would probably click that every now and then.
To be fair (and I am not trying to defend these 'Slick guys'), sometimes there is a lot more happening behind closed doors that a PM might let on. Fights for resources, maintaining a current team, hiring, ownership, etc.
Oftentimes those people trying to portray a 'Slick' exterior are doing so due to the need to portray a sense of success/confidence for their team, to make sure they retain their existing budget and that the team doesn't get moved to other projects or terminated entirely.
PM @ a well funded AI startup. Previously PM at Netflix. Management consultant prior.
Undergraduate degree in Psychology. Background in marketing/business and customer acquisition. I learned enough programming to automate my marketing activities, and found that I liked driving a roadmap more than I liked acquiring customers.
-Communication and conflict resolution skills are key. You are in a role where you must drive influence without having any direct reports. This means effective, articulate communication skills are required. Know how your voice needs to change between communication to engineering versus communication to an executive or board member.
-At Netflix, I was often told my job was to add clarity. Add clarity to a technical specifications document. Add clarity to the marketing teams understanding of a product feature. The best PMs are able to consolidate their understanding of a 35 page technical document into two sentences.
-Market sizing and back of the envelope calculations. Know how large the market is for your product. How much more can you charge for your product if you add X feature? How long is X feature going to take in engineering cycles? Is this the best way to spend your engineering resources? In my daily routine, I probably make 10 calculations like this and have a response ready for either our product director, CEO, board member, or customer.
-Financial modeling. I've found that modeling skills are absolutely key - know how to model out customer lifetime value, churn rates, and cash flow. You should be prepared to be a 'mini CFO,' because at the end of the day, you are asking for more resources from your executive suite, and are best off making those requests in CFO format.
-Know your technology. Know what is possible and know how to articulate requirements that speak to your technology. This is why there is often a technical barrier for PMs - you have to know how things work, and what is physically possible versus cost prohibitively impossible. This doesn't mean you need to know how to code - but that is helpful. Know source control and developer operations processes. Know how to plan for scale. Know how to recognize elegant solutions for difficult problems, and reward your engineering team for failing spectacularly.
-Finally - be humble and be accountable. It is always your fault, because you are accountable for the success of your product. Don't throw your engineering team into the middle of a sh*tstorm of management politics - be their umbrella. Don't blame customers, politics, or resources. It's always your fault. Find a way to fix it.
>-Finally - be humble and be accountable. It is always your fault, because you are accountable for the success of your product. Don't throw your engineering team into the middle of a sh*tstorm of management politics - be their umbrella. Don't blame customers, politics, or resources. It's always your fault. Find a way to fix it.
Man, I would love to work for someone that actually strives to do this. Awesome.
Were you at any organisations where they did 360° feedback? What kind of feedback did you receive from developers or designers from your team? What I'm trying to understand is how are organisations ensuring that PMs and team are not in disconnect from how things really are. I know that's a different realm but just trying to understand how did you find what to improve or learn to be a better PM?
Netflix and my current organization religiously practice 360 degree feedback. For my own projects, I practice 360 degree feedback as well. I don't think a team or organization can succeed if employees cannot speak to each other candidly about performance.
My typical feedback from engineering:
1) I state resolutions of a problem without clearly defining the problem.
This was/is my biggest failure as a product manager and is something I work on daily. I enjoy the 'fun' of solving problems but respect that my job is not to solve the problem. My job is to understand the market, define customer and their needs, and create requirements that need to be met to resolve those customer needs.
2) I over-engineer. I like to solve problems with complex, scalable, 'sexy' solutions. At Netflix, my team built a real time marketing analytics platform that used kafka/spark/elasticsearch and an enormous cluster to aggregate marketing data from 5+ marketing platforms. The client was built in angular/d3 and returned aggregations on 1B+ rows of data in < 100ms.
We were so invested in scale and performance that minor changes to the underlying schema (which happened often, as marketing priorities shifted) required a lot of work. This was a huge over engineering mistake on my behalf.
3) I can come off as patronizing. In an effort to describe a problem space or market, my tone has been perceived as patronizing.
4) I do not practice enough active listening. I end up driving conversations and do not make people feel heard.
Being humble and asking for feedback is the best way to learn to be a better PM. Of the PMs I've seen rise(and fall) through the ranks of management, I have generally found that humility, integrity/accountability, and communication skills are the most correlated with success.
What a wonderful and candid reply. Those 4 things are common feedbacks as a PM. It's a role where it can be easy to fool yourself that you think you already know everything. I believe a lot of PMs (myself included) often/always need to continuously improve on. But few would be as open and receptive to speaking of them, or ready to actively work on being more humble, more accountable.
This is so awesome, wasn't expecting such detailed response! I think we all have some of these flaws but having channels to have this feedback and to have open discussions without any ego or taking it personally makes the best teams. Currently I'm not in such a team but I was in past and wish to find one soon!
Thanks for sharing. How are you guys organized at Netflix? Is there a distinction between platform product managers and solution product managers? From outside it looks like the world is divided as Content led by Ted Sarandos and Platform led by Neil Hunt.
when I asked this question I first expected a lot of answers in the lines of your "Financial Modeling" part. Funnily, you are the only one that's named it so far. Anyhow all the answers have been so incredibly enriching...
Advertisers aim to optimize expected revenue over cost within a 'payback period', which is typically 1-3 years for large advertisers. This is calculated through customer acquisition cost, retention, and incrementality (the probability of an ad causing conversion).
Only advertisers themselves can effectively calculate incrementality due to access to their specific conversion/retention metrics. This incrementality, along with the optimal mix of channels and ad spend, is the ongoing challenge for sophisticated advertisers. It involves multi-objective optimization across millions of ad assets, campaigns, and targeting criteria.
Privacy regulations since 2015 and subsequent laws like GDPR and CCPA have led to more reliance on probabilistic modeling for targeting. The entire pipeline of targeting, engagement, conversion, and retention forecasts are now based on probabilistic models.
While ad networks offer simplified scaling solutions like 'target roas' and 'campaign budget optimization', they're more useful for average advertisers with limited internal resources - eg, they can't justify hiring ML SWEs, quant traders, technical PMs, etc.
Advertising has become even more data-driven and arbitrage gains for sophistication have increased. Profound gains can be made with investments in marketing and forecasting science, similar to the operations of a quant trading firm.
Source: I've managed $10B+ through automated ad spend systems since 2012.