The problem is that humans use this as a coping mechanism for things they don't understand: I don't understand why the printer doesn't work, so I give it a mind of its own.
This is harmless for inconsequential stuff like a chair, but when it's an LLM, people should at least understand it's behavior so they don't get trapped. That means not trusting it with advice meant for the user or on things it has no concept of, like time or self-introspection (people ask the LLM after it acted, "Why did you delete my database?" when it has limited understanding of its own processing, so it falls back to, "You're right, I deleted the database. Here's what I did wrong: ... This is an irrecoverable mistake, blah, blah, blah..."
Humans have extremely limited understanding of their own processing?
When you ask a human why they did something wrong, they usually confabulate an answer as well.
Human conscious introspection doesn't extend to actual processing, it is limited at best to recollection of internal experience leading up to the point in question. That internal experience in turn represents but a tiny fraction of what actually happens in the brain and does so on a pretty abstract level only.
"Anthropomorphizing" is a red herring. Humans understand themselves so insufficiently, they can't claim reasonably founded judgement either way.
When you don't know what you're doing, you probably shouldn't be doing it.
Instead of saying, "You gave me the access permissions and failed to add any guardrails, so effectively you deleted the database using me as the tool."
But your typical LLM doesn't even have enough grasp to say that. Which still doesn't stop the believers from insisting that it has genuine intelligence and consciousness.
No, the cache is a few GB large for most usual context sizes. It depends on model architecture, but if you take Gemma 4 31B at 256K context length, it takes 11.6GB of cache
note: I picked the values from a blog and they may be innacurate, but in pretty much all model the KV cache is very large, it's probably even larger in Claude.
To extend your point: it's not really the storage costs of the size of the cache that's the issue (server-side SSD storage of a few GB isn't expensive), it's the fact that all that data must be moved quickly onto a GPU in a system in which the main constraint is precisely GPU memory bandwidth. That is ultimately the main cost of the cache. If the only cost was keeping a few 10s of GB sitting around on their servers, Anthropic wouldn't need to charge nearly as much as they do for it.
That cost that you're talking about doesn't change based on how long the session is idle. No matter what happens they're storing that state and bring it back at some point, the only difference is how long it's stored out of GPU between requests.
Are you sure about that? They charge $6.25 / MTok for 5m TTL cache writes and $10 / MTok for 1hr TTL writes for Opus. Unless you believe Anthropic is dramatically inflating the price of the 1hr TTL, that implies that there is some meaningful cost for longer caches and the numbers are such that it's not just the cost of SSD storage or something. Obviously the details are secret but if I was to guess, I'd say the 5m cache is stored closer to the GPU or even on a GPU, whereas the 1hr cache is further away and costs more to move onto the GPU. Or some other plausible story - you can invent your own!
Storing on GPU would be the absolute dumbest thing they could do. Locking up the GPU memory for a full hour while waiting for someone else to make a request would result in essentially no GPU memory being available pretty rapidly. This type of caching is available from the cloud providers as well, and it isn't tied to a single session or GPU.
> Storing on GPU would be the absolute dumbest thing they could do
No. It’s not dumb. There will be multiple cache tiers in use, with the fastest and most expensive being on-GPU VRAM with cache-aware routing to specific GPUs and then progressive eviction to CPU ram and perhaps SSD after that. That is how vLLM works as you can see if you look it up, and you can find plenty of information on the multiple tiers approach from inference providers e.g. the new Inference Engineering book by Philip Kiely.
You are likely correct that the 1hr cached data probably mostly doesn’t live on GPU (although it will depend on capacity, they will keep it there as long as they can and then evict with an LRU policy). But I already said that in my last post.
Yesterday I was playing around with Gemma4 26B A4B with a 3 bit quant and sizing it for my 16GB 9070XT:
Total VRAM: 16GB
Model: ~12GB
128k context size: ~3.9GB
At least I'm pretty sure I landed on 128k... might have been 64k. Regardless, you can see the massive weight (ha) of the meager context size (at least compared to frontier models).
On HN, I often see comments like this, complaining about Cloudflare blocking access to pages. It makes me wonder if it’s due to a particular setup that triggers bot detection – like Tor or no-JS – that HN readers often use, or if Cloudflare has too many false positives.
I think it's aggressive user profiling, so anyone with a hint of privacy is not welcomed. I can't imagine this getting any better with Chrome MCP and other tools.
I’m more hopeful that MIDI completion/in-filling models will be easier for musicians to control and use. But right now, the most popular tools are things like Suno, where you barely have any control and it spits out an entire, possibly mediocre song. It’s the same vein as ChatGPT image generation vs. Stable Diffusion, where you can do much more controllable inpaints with the latter.
> Nicotine in tobacco can form carcinogenic tobacco-specific nitrosamines through a nitrosation reaction. This occurs mostly in the curing and processing of tobacco. However, nicotine in the mouth and stomach can react to form N-nitrosonornicotine, a known type 1 carcinogen, suggesting that consumption of non-tobacco forms of nicotine may still play a role in carcinogenesis
The dose in urine is 1-3% of that of cigarette smokers so it is a significant, order of magnitude decrease in risk based on the paper another GP has posted below. In the mouth the levels also seem to be an order of magnitude lower than cigarette smokers (though similar in a majority of cases). Those are relatively acceptable risks for a vice I would think.
This is harmless for inconsequential stuff like a chair, but when it's an LLM, people should at least understand it's behavior so they don't get trapped. That means not trusting it with advice meant for the user or on things it has no concept of, like time or self-introspection (people ask the LLM after it acted, "Why did you delete my database?" when it has limited understanding of its own processing, so it falls back to, "You're right, I deleted the database. Here's what I did wrong: ... This is an irrecoverable mistake, blah, blah, blah..."
reply