People like the horse joke, but it works with computers in general. The best argument in favor of computers in the 80s was to save paper, otherwise they were expensive and overcomplicated.
If you are correcting my use of millions to trillions: I was refering to the author himself, who writes like this giant AI bubble is pushing him forward to the edge of the cliff as people keep believing in it, and he is desperately trying to get the bubble to shrink or he'll fall off and die. Methaphorically.
But why does he act or feel that way? Let the trillions be lost, it's just how hypes, bubbles and the stock market in general work.
That would be nice, but I'm more worried they won't collapse because they'll succeed at shoving their snake oil down the throats of enough big players to ensure their survival.
Google solved a real problem. They indexed the web and made search work, and they did it very cheaply. So cheaply, in fact, that they could give their service away to users and monetize it with ads. LLMs are not like this. They're both extremely expensive to run and they don't do anything truly valuable--there's no killer app. So how exactly is OpenAI or their ilk (or for that matter the rest of us) supposed to use these things to make money?
This is the only question, and the fact it's still an open question just screams "hype bubble". My bet is this AI stuff goes the way of the NFT.
I'd take that bet. Google offers a very expensive service for free, but is able to monetize it with ads. Sometimes connecting users to companies is what users actually want. But Google has this problem that since the service is free, its users feel entitled to everything for free. They can't just go and charge people what it costs to run a Google search.
OpenAI doesn't have this problem. ChatGPT has a free level to get you hooked, but it's restricted. So a lot of users pay them $20 or $200 or some other amount per month to use their service. So how OpenAI makes money is by selling access to their service. What you do with it is up to you, but their value proposition is simple. Pay us to get more/better access to our service.
How much it costs them to operate the service is a secret known only to them. There are a lot of very very educated guesses, but they're just guesses. After the VC money runs out they'll have to charge more than it costs to provide the service to stay afloat, and then we'll see. $20/month for ChatGPT plus is the $1 Uber that got people hooked. There's already a $200/month tier.
Whether OpenAI, specifically, will be standing in 20 years, only time will tell. But by this point it should be obvious that there's something to this LLM thing. Even if the product doesn't get any better than it is today, it'll still take 5-10 years for its effects to reverberate through society.
The killer app is LLM-accelerated programming. Sure, it doesn't work for all domains and it can't do everything, but even if the only thing it's good for is creating JavaScript react CRUD apps, well, there are a lot of those out there, and they're not actually limited to that. And since tool use means they can generate code and compile it and test that it works, it's possible to generate datasets for other languages and libraries, the only question is which ones is it worth it for.
It might not help at all in your line of work, but a friend who does contracting is able to use LLMs to cut the time it takes him to do a specific kind of job in half, if not more,
enabling him to take on twice as many clients and make more money. For him it would still worth it even at 100x the current price. thankfully competition means it'll take a while before it's that expensive.
I've heard from insiders that AWS Nova and Google Gemini - both incredibly cheap - are still charging more for inference than they spend on the server costs to run a query. Since those are among the cheapest models I expect this is true of OpenAI and Anthropic as well.
The subsidies are going to the training costs. I don't know if any model is running at a profit once training/research costs are included.
As a society we choose to let the excess wealth pile up into the hands of people that are investing to bring about their own utopia.
If we're stretching, we can talk about opportunity cost. But the people spending and creating the "bubble" don't have better opportunities. They're not nations that see a ROI on things like transportation infrastructure or literacy.
So unless the discussion is taken more broadly and higher taxes are on the table, there really isn't a cost or subsidy imo.
This. IIUC to serve an LLM is to perform an O(n^2) computation on the model weights for every single character of user input. These models are 40+GB so that means I need to provision about 40GB RAM per concurrent user and perform hundreds of TB worth of computations per query.
How much would I have to charge for this? Are there any products where the users would actually get enough value out of it to pay what it costs?
Compare to the cost of a user session in a normal database backed web app. Even if that session fans out thousands of backend RPCs across a hundred services, each of those calls executes in milliseconds and requires only a fraction of the LLM's RAM. So I can support thousands of concurrent users per node instead of one.
> IIUC to serve an LLM is to perform an O(n^2) computation on the model weights for every single character of user input.
The computations are not O(n^2) in terms of model weights (parameters), but linear. If it were quadratic, the number would be ludicrously large. Like, "it'll take thousands of years to process a single token" large.
(The classic transformers are quadratic on the context length, but that's a much smaller number. And it seems pretty obvious from the increases in context lengths that this is no longer the case in frontier models.)
> These models are 40+GB so that means I need to provision about 40GB RAM per concurrent user
The parameters are static, not mutated during the query. That memory can be shared between the concurrent users. The non-shared per-query memory usage is vastly smaller.
> How much would I have to charge for this?
Empirically, as little as 0.00001 cents per token.
For context, the Bing search API costs 2.5 cents per query.
Interesting. There's obviously been a precipitous drop in the sticker price, but has there really been a concomitant efficiency increase? It's hard to believe the sticker price these companies are charging has anything to do with reality given how massively they're subsidized (free Azure compute, billions upon billions in cash, etc). Is this efficiency trend real? Do you know of any data demonstrating it?
I have personal anecdotal evidence that they're getting more efficient: I've had the same 64GB M2 laptop for three years now. Back in March 2023 it could just about run LLaMA 1, a rubbish model. Today I'm running Mistral Small 3 on the same hardware and it's giving me a March-2023-GPT-4-era experience and using just 12GB of RAM.
People who I trust in this space have consistently and credibly talked about these constant efficiency gains. I don't think this is a case of selling compute for less than it costs to run.
People are comparing the current rush to the investments made in the early days of the internet while (purposely?) forgetting how expensive access to it was back then. Not saying that AI companies should make a profit today, but I don't see or hear that AI usage is becoming essentials in any way or form.
Yeah that's the big problem. The Internet (e-commerce, specifically) is an obviously good idea. Technologies which facilitate it are profitable because they participate in an ecosystem which is self sustaining. Brick and mortar businesses have something to gain by investing in an online presence. As far as I can tell, there's nothing similar with AI. The speech to text technology in my phone that I'm using right now to write this post is cool but it's not a killer app in the same way that an online shopping cart is.
This is the crux. A cool thing has been invented, with real usages. Unfortunately, it's cost hundreds of billions of dollars and it has absolutely zero hope of making the trillions needed to justify that.
Now someone will respond about how it's just a stepping stone, and how the billions are justified by _something completely imaginary, and not invented yet, and maybe not ever_ e.g. agents.
>it's cost hundreds of billions of dollars and it has absolutely zero hope of making the trillions needed to justify that.
The BigTech companies have been flush with liquidity and poured those hundreds of billions into the promising tech, and as result we got a wonderful new technology. There is not much need for those trillions in return - just look at liquidity positions of those companies, they are just fine. If those trillions come in eventually - even better.
>There is not much need for those trillions in return
Whilst you are correct that big tech cos do not need the return to survive, that's not how public markets work at all, and thus not how the incentives for those in charge of the companies work, and so making you actually wrong.
If i were wrong, those companies would be distributing that cash to shareholders instead of chasing any promise of any big chance.
If investment in AI don't pan out (i do think that it will pan out, and those trillions will come) then those companies would just pour even more billions into whatever big thing/promise would come next. Rinse and repeat. Because some of those things do generate tremendous returns, and thus not playing that game is what really constitute true loss of money.
US right now is run by someone whose explicit promises, if actually implemented, have an obvious immedidiate 13-14% reduction in GDP — literally, never mind side effects, I'm not counting any businesses losing confidence in the idea that America is a place to invest, this is just direct impact.
DOGE + deportation by themselves do most of that percentage. The tariffs are a rounding error in comparison, but still bad on the kind of scale that gets normal politicians kicked out.
Nobody has ever been punished for choosing IBM. It’s the same story here. Nobody is going to blame them for following the zeitgeist, but you bet they’d be punished if they didn’t and it doesn’t pan out.
The whole thing is like bitcoin. There’s too many people that benefit from maintaining the collective illusion.
cash on hands GOOG - 100B, AMZN - 80B, FB - 70B, and their core businesses are basically printing money, so they pretty much do have to invest into new things. If somebody sees a multi-billion dollar sink better than AI right now ...
Ideally it was thought to have shortened a very expensive war, and may have prevented the USSR from taking over Europe by leveraging its unquestioned postwar conventional forces advantage.
I don't know why it is so hard to understand. I mean money doesn't really exist without a government[0] and while government plays a role in the market and economy, this role is VERY different than that of a business. A government isn't trying to "make money", is isn't trying to make investors happy, and it certainty can't take existential risks that could make "the company" go bankrupt (or it shouldn't lol).
But I do think (and better understand) there is a failure to understand this at a higher abstraction. One part is simply "money is a proxy." This is an uncontestable fact. But one must ask "proxy for what?" and I think people only accept the naive simple answer. Unfortunately, this "is a proxy" concept is extremely generalization. Everything is an estimation, everything is an approximation, and most things are realistically intractable. We use sibling problems or similar problems to work with that are concrete, but there are always assumptions made and ignoring these can have disastrous consequences. Approximations are good (they're necessary even) but the more advanced a {topic,field,civilization,etc} gets, the more important it is to include higher order terms. Frankly, I don't think humans were built for that (though by some miracle we have the capacity to deal with it).
My partner and her dad are both economists, and one thing I've learned is that what many people think are "economics questions" are actually "business questions". I think a story from her dad makes this extremely clear. A government agency hired him to look at the cost benefit analysis of some stuff (like building a few hospitals and some other unambiguously beneficial institutions), and when he presented everyone was happy but had a final question "should we build them?" The answer? "That's not the role of an economist." The reason for this is because money can't actually be accurately attributed to these things. You can project monetary costs for construction, staffing, and bills, and you can make projections about how many people this will benefit, how it can reduce burdens elsewhere, and as well as make /some/ projections about potential cost savings. But you can't answer "should you." Because the weight of these values is not something that can be codified with any data. It is an importance determined by the public and more realistically their representatives. Very few times can you give a strong answer to a question like "should we build a new hospital" and essentially in only the extreme cases. I'll give another example. In my town there was an ER that was closed due to budget constraints. This ER was across the street to the local university, which students represent ~15% of the population. The next nearest ER? A 15 minute ambulance ride away and in the next town over. Did the city save money? Yes. Did the sister city's ER become even busier? Also yes. Did people lose access to medicine? Yes. Did people die? Also yes. Have economists put a price on human life? Also yes, but they are very clear that this is not a real life and a very naive assumptions[1]. It is helpful in the same way drawing random squiggles on a board can help a conversation. Any squiggles can really be drawn but the existence of _something_ helps create some point to start from.
[0] okay crypto bros, you're not wrong but low volatility is critical as well as some other aspects. Let's not get off topic
The profit was made by the private sector in supplying goods to the program. Today, private companies do a lot and earn a lot of money from stockpile maintenance.
The Manhattan Project was driven by the U.S. Government, which doesn't need a VC-tier return. The entire business model of VCs is based on the idea that they'll have the occasional 100x return, and if none of the AI companies do that it would destroy the VC model.
Doesn't matter. The Manhattan project was a breakthrough in fundamental science that changed the world. Current generative AI are a solid degree improvement on previous technology that is not remotely as big a leap as the amount of money poured into it assumes it to be.
> I use Cursor sometimes, and VSCode + Continue with llama.cpp, and it's great. That's not worth billions. It's definitely not worth trillions.
That seems like a suspect claim. If you're saying that you, personally, cannot create billions of dollars in value with Cursor & friends that is certainly true - but you are in no position to make a judgement call about where the cap on value creation is for the LLM market is worth based on your personal use cases. LLMs don't just do code completion. We really can't estimate how much potential value is being created without doing some serious data diving and studying of cases.
A better argument would be that the DeepSeek experience suggests these companies have no moat and therefore no way to earn a return on capital. But LLMs are probably going to generate at least trillions of dollars in value because they're on par or ahead of Wikipedia and Google for answering many queries then they also have hundreds of ancillary uses like answering medical questions at weird hours or creative/professional writing.
It's possible to grow an economy by trillions of real value without any actor being able to extract that as a profit or it even showing up in the books as money.
Consider that Wikipedia is much bigger than Encyclopedia Britanica, but because it is given away to everyone for free, it is not counted as E.B.'s max sale price ($2900 in 1989?) times the world's internet connected population (5.6e9?) — $16 trillion.
AI, regardless of value, are priced at the marginal cost to reproduce weights or run inference depending on which you care about.
But I do mean "reproduce" not "invent" — it doesn't matter if DeepSeek's "a few million" was only possible because they benefited from published research, it just matters that they could.
And if the hardware is the bottleneck for inference, that profit goes to the hardware manufacturer, not to the top ten companies who made models.
Where are you getting this from? Outside of o3, every AI provider's API is super cheap, with most productive queries I do coming in under 2c. We have no reason to believe any of them are selling API requests at a loss. I think <2c per query hardly counts as "quite expensive".
The reasoning people have for them selling API requests at a loss is simply their financial statements. Anthropic burned $3B this year. ChatGPT lost $5B. Microsoft has spent $19B on AI and Google has spent close to $50B. Given that revenue for the market leader ChatGPT is $3.7B, it's safe to say that they're losing massive amounts of money.
These companies are heavily subsidized by investors and their cloud service providers (like Microsoft and Google) in an attempt to gain market share. It might actually work - but this situation, where a product is sold under cost to drum up usage and build market share, with the intent to gain a monopoly and raise prices later on - is sort of the definition of a bubble, and is exactly how the mobile app bubble, the dot-com bubble, and previous AI bubbles have played out.
Not sure if it matters at this point. There will need to be many more rounds of CapEx to realize the promises that have been put forth about these models.
The implication would be that those API requests are being sold at a loss. Amodei wrote in January that Claude 3.5 Sonnet was trained for only a few $10Ms, but Anthropic has been losing billions.
That would be a killer for the current and near future generations of LLM as a business. If they are having to pay many times in compute what they are able to get for the API use (due to open models being near comparable?), then you definitely can't "make up for it in volume".
How? I get that many devs like using them for writing code. Personally I don't, but maybe someday someone will invent a UX for this that I don't despise, and I could be convinced.
So what? That's a tiny market. Where in the landscape of b2b and b2c software do LLMs actually find market fit? Do you have even one example? All the ideas I've heard so far are either science fiction (just wait any day now we'll be able to...) or just garbage (natural language queries instead of SQL). What is this shit for?
Anecdotally, almost every day I’ll overhear conversations at my local coffee shop of non-developers gushing about how much ChatGPT has revolutionized their work: church workers for writing bulletins and sermons, small business owners for writing loan applications or questions about taxes, writers using it for proofreading, etc. And this is small town Colorado.
Not since the advent of Google have I heard people rave so much about the usefulness of a new technology.
These are not the sort of uses we need to make this thing valuable. To be worthwhile it needs to add value to existing products. Can it do that meaningfully well? If not it's nothing more than a curiosity.
To make money though it just needs to have a large or important audience and a means of convincing people to think, want, or do things that people with money will pay to make people think, want or do.
Can you get enough revenue from ads to pay the cost of serving LLM queries? Has anyone demonstrated this is a viable business yet?
A related question: has anyone figured out how to monetize LLM input? When a user issues a Google search query they're donating extremely valuable data to Google that can be used to target relevant ads to that user. Is anyone doing this successfully with LLM prompt text?
I bet Google is utilizing the value of the LLM input prompts with close to the same efficiency they are monetizing search. I that case, there are two questions -- 1) will LLM overtake search? and 2) can anyone beat Google at monetizing these inputs? I think the answer to both is no. Google already has a wide experience lead monetizing queries. And personally, I'd rather have a search engine that does a better job of excluding spam without having to worry whether or not it's making stuff up. Kagi has a better search than any of the LLMs (except for local results like restaurants/maps).
My company uses them for a fuckton of things that were previously too intractable for static logic to work (because humans are involved).
This is mostly in the realm of augmented customer support (e.g. customer says something, and the support agent immediately gets the summarized answer on their screen)
It’s nothing that can’t be done without, but when the whole problem can be simplified to “write a good prompt” a lot of use cases are suddenly within reach.
It’s a question if they’ll keep it around when they realize it doesn’t always quite work, but at least right now MS is making good money off of it.
LLMs are incredible at editing my writing. Every email I write is improved by LLMs. My executive summaries are improved by LLMs. It wont be long until every single office worker is using LLMs as an integral part of their daily stack, people just have to try it and theyll see how useful it is for writing.
Microsoft turned itself into a trillion dollar company off the back of enterprise SAAS products and LLMs are among the most useful.
Various minor thing so far. For example I heard about ChatGPT being evaluated as a tool for providing answers for patients in therapy. ChatGPT answers were evaluated as more empathetic, more human and more aligned with guidelines of therapy than answers given by human therapists.
Providing companionship to lonely people is another potential market.
It's not as good as people at solving problems yet but it's already better than humans at bullshiting them.
Are people actually satisfied by that? I personally find "chatting" with an LLM grating and dissatisfying because it often makes very obvious and incongruous errors, and it can't reason. It has no logical abilities at all, really. I think you're really underestimating what a therapist actually does, and what human communication actually is. It's more than word patterns.
I could see this being useful in a "dark pattern" sense, but only if it's incredibly cheap, to increase the cost to the user of engaging with customer support. If you have to argue with the LLM for an hour before being connected to an actual person who can help you, then very few calls will make it to the support staff and you can therefore have a much smaller team. But that only works if you hate your users.
Subjective evaluation of "humanity" and "empathy" in responses is much less important than clinical outcome. I don't think an online chat with a nebulous entity will ever be as beneficial as interactions that can, at least occasionally, be in-person. Especially as the trust of online conversations degrade. Erosion of trust online seems like a major negative consequence of all the generative AI slop (LLM or otherwise).
Clinical outcome of humans doing therapy would be better if for some reason doing therapy worse (less according to taught guidelines) was better. But, sure, we can wait for another research or follow up. It might be true. Therapy has dismal outcomes anyways and the outcomes are mostly independent of which theoretical framework the therapy is done according to. It might be the case that the only value in therapy is human connection that AI fails to simulate. But it seem that for some people it simulates connection pretty well.
> The article is about how the economics of the LLM market is making all tech look bad.
No, it's not. The first half of the article talks about how useless the actual product is, how the only reason we hear about it is because the media loves to talk about it.
