Working on benchmark arena for AI agents with my wife.
We grab interesting business problems, turn them into fun challenges for hundreds of AI engineers to find the best architecture for. Insights are shared back with the community.
It is a fun learning process with unexpected scaling challenges.
I wonder if projects which are anti-AI could place such identifiers surreptitiously into docs or commits as a way to sabotage people using Claude Code. Your project isn't going to get many AI PRs if just cloning your project wiped out their quota.
There's no separation between parts of the prompt. You sneak that text in, anywhere, and it'll work. Whether Anthropic is using a regex or some LLM to detect the mentions of OpenClaw doesn't even matter.
> Your project isn't going to get many AI PRs if just cloning your project wiped out their quota.
With how many projects automatically AI-review PRs, they're just sitting ducks. You don't even need to hide it, put it clear and center and there's your denial of service.
You don't even need to put it in a project, put it in all your blog posts as invisible (white font white background) text, and if Claude winds up reading your website as part of a research task, you basically bricked someone's Claude session.
Because AI is a new product category in tech, and every single new product category in tech always, no exceptions, insists on learning nothing from history, and so the dumb shit is repeated until they learn their own lessons.
I am almost 40, and I have seen the same pattern play out several times now, it’s always the same.
> every single new product category in tech always, no exceptions, insists on learning nothing from history,
I've worked in a bunch of industries and places over the years, and this is not just a tech thing. Like, there's a reason that saving a day in the library with a week in the lab is a pretty famous saying.
Reminds me of the time a former employer which shall remain nameless paid a Senior Developer to spend an entire year coding something a $15,000 license from the maintainers of the original library would have given them. So lets spend 6 figures to save 15 grand or whatever.
This was a CTO burning funds, and that does not even cover the maintenance costs, especially as the original library changes and becomes drastically more modern.
>
Reminds me of the time a former employer which shall remain nameless paid a Senior Developer to spend an entire year coding something a $15,000 license from the maintainers of the original library would have given them. So lets spend 6 figures to save 15 grand or whatever.
You argumentation assumes that the goal was saving money.
On the other hand, if the company's goal is to become a little bit more independent of this library (and their licensing fees), this approach often makes a lot of sense.
I just used this a few weeks ago, except it was time not money. And I'm on my fourth implementation because nobody wants to stop and actually have a plan.
what if kings attacking and burning down libraries of advanced civilizations (Nalanda, Alexandria) is a way for humans to reset the world's knowledge, because we got bored of our achievements and want to start from scratch ?
Ageism is definitely part of it, but most people just don't seem to care to learn in general, and of course the incentives are against it.
They'd rather treat the general version of Greenspun's 10th rule as a commandment, and create a new, ad hoc, informally-specified, bug-ridden, slow implementation of some fraction of whatever already addresses the requirement, than learn about how to use some existing tool that they don't already know.
One of my favorite examples is a company that home-rolled their own version of (a subset of) Kubernetes, ending up with a fabulously fragile monstrosity that none of the devs want to touch any more, and those who do quickly regret it.
How does BEAM renew my certificates, configure reverse-proxies, mount networked storage volumes to whichever node a given process is running on and handle cronjobs, disk pressure and secrets?
I sure hope it doesn't involve a bunch of shell scripts to create a new, ad hoc, informally-specified, bug-ridden...
Nah Kubernetes is a systems level, language agnostic (at least doesn’t force you to run Golang workloads) variant of J2EE. It’s basically modern day Websphere
1. there's too much to learn and know, and the cs courseware and interviews are all about algorithmic complexity, rather than business setup and operations. same with "how do you raise money" vs "how do you make a great customer experience"
2. the market rewards building the new functionality, not building all the standard chunks required to run a business
if anthropic had great customer support built out, but no model and no claude-code, theyd be much worse than they are now having stuff people want but lacking the ability to serve it
---
clearly theres still plenty of b2b startup opportunities standardizing how all these things work, so that business can focus on their actual business rather than recreating all the basics
There can be reasons for reinventing something, both good and bad. But I was more talking about the cases where people regiment something because they either didn't bother to do the slightest bit of research into what was already available, or because they didn't care and just wanted to do it themselves, inflicting their inevitably poor solution on everyone else.
> Because AI is a new product category in tech, and every single new product category in tech always, no exceptions, insists on learning nothing from history, and so the dumb shit is repeated until they learn their own lessons.
I'm only half a decade behind you, and I agree. Sad to see really, these are people who work really hard, but I think they are too focused on the algos and nobody is hiring experienced back-end and application builders.
What's the chance that it is market motivated? That the companies most likely to succeed are those willing to break the rules (this isn't to say that breaking the rules makes one likely to succeed, you have to break the right rules and not the wrong ones, and that distinction is often times unknown til after the fact).
This might mean that the companies that we see explode in popularity are those whose cultures are already biased in ways that don't consider negative outcomes, as the companies that did consider them already excluded themselves from exploding in the market (they might still be entirely successful startups, but at a vastly smaller scale of success).
It is absolutely market motivated, by the investor market. You can raise a great deal of capital by simply making exaggerated promises, then doing the minimum effort to just about achieve it.
Every time something new comes along, people go "we are the new hotness, all those pesky lessons those old guys have learned over the last 200-or-so years don't apply to us." It applies to tech. It applies to crypto. It applies to political revolutions. Every time, it ends the same way (with the political revolutions inevitably being a lot more deadly).
I am a little over 50 and I have also seen the same pattern play out. It's incredible.
Lots of things were the Hot New Things That Will Change Everything, like VLIW processors, transputers before that, no doubt others. Perceptrons! Oh wait they can't do XOR functions, well how about Neural Networks? Too complex! Tell you what then, Fuzzy Logic, it'll power everything from washing machines to self-driving cars! Now we're at LLMs that are just neural network-powered Eliza bots that pirate everything like you did the week you first discovered Torrentleech.
Some things have stuck around, like OOP and RISC processors. Others like Quantum Computing are - like Iran's nuclear weapons program - just weeks away from blowing away everything we know, for the past 40 years or so.
Everything runs on relational databases on thumping great Unix boxes and that's unlikely to ever change.
> and every single new product category in tech always, no exceptions, insists on learning nothing from history
I dunno. Maybe what they learned is that SaaS products have a familiar flow. They push as many restrictions as they can get away with to protect their moat until they do something that causes noticeable dip in subscriptions. Then they issue a disingenuous mea culpa and find another way to protect their walled garden.
Now users on the other hand. Might they be the ones who haven't learned from being jerked around in exactly the same way by SaaS providers for the past 20+ years?
I saw a similar shift at a another company when the VCs brought in new, incompetent management after it had gotten successful and started piling on industrial levels of pressure.
My bet would be that a lot of the ICs and managers who made anthropic what it is have been sidelined and investor yes-men with puffy resumes are now running things while investors panicked about high interest rates breathe down their neck.
Those are dark patterns and people are not aware of them. It is an external actor trying to take control of your agent.
I don't think it's necessarily wrong to have those prompts, but it is if it's hidden or obscured. Intent matters a lot here. Which the response to name shaming (and how you name shame) is actually the important part. Getting overly defensive is not the appropriate response. Adding clarity and being more transparent about why such a decision was made is the correct response. We're all bumbling idiots and do stupid stuff. But there's a huge difference between being dumb and malicious, even if the outcome is the same
FYI this does not work for CTF challenges at least - I’ve seen a lot of rev/pwn challenges try to add magic refusal strings/prompt hijacking and models really don’t give a damn.
Frankly if a project asks for no AI and you try to use AI for it, then you kinda deserve this. Calling the inclusion of this sort of thing "smuggling" is placing the blame in the wrong spot
I used the term "smuggling" in the casual sense of hiding something. I have edited it to "place such identifiers surreptitiously" to avoid making whatever implication appears to have been taken.
> I wonder how long these sorts of games will play before the law applies itself.
Perhaps roughly as long as the law turns a blind eye to AI corps flagrantly violating the attribution requirements of software licenses that apply to their training data, as well as basically ignoring other copyright requirements at scale. Fair use, my eye.
It's Antropic defrauding people here, the person using it for fighting anti-social behavior (or even a troll doing the anti-social behavior themselves) isn't guilty of it.
if someone is trying to use LLM tools in a project that explicitly forbids the use of LLM tools, they are not innocent.
if someone is blinding slurping up content to feed to LLMs, without checking to see if a particular source is OK with that, they are arguably not innocent either.
Neither situation is analogous to a booby-trapped shotgun door blowing off the face of a would-be burglar.
I'm not leaving boody traps. I have the right to talk about OpenClaw or even to write the anti antropic string. I didn't delete you token usage or charge you extra boxes. Antropic did.
If tomorrow Antropic decide to charge you extra if you interact with someone who talked badly about them, I'm still in my right to talk shit about them.
This is the same logic of 'not a booby trap' booby trap,s which sometimes do work out in the favor of the one setting them if they weren't too open about it. If your commit message is that you are talking about OpenClaw just to booby trap your repo, then I suspect it wouldn't fly, where as if you gave it some plausible deniability, a lawyer would be able to get any suit or charges dismissed.
This is all under the assumption we eventually live in a world where booby trapping repositories becomes a legal issue. On one hand that feels silly. On the other hand, we have had far less sensible cases make it to court and there is a small kernel of similarity which the legal system might latch onto.
If someone doesn't want you to use AI on their repository, they state it.
And if they want to "booby trap" (Antropic logic), them it's they right, you have been warned.
I can't see how you rights to use AI is prevalent on the right of anybody to write the string "OpenClaw" or any string forbidden by your AI provider.
Seriously, if the author hides it and trick your AI agent to check it, well maybe. But otherwise, it's not even a question.
This is extremely naive. If you are in Germany and I am in the US and you get a default judgement against me (which would cost you money to get), good luck getting it enforced internationally. Hint: it's way, way harder than you think.
I don't think the GP is calling contributor guideline restrictions a form of DRM.
I think the GP is focusing on:
> I guess we're giving up on the idea that you're free to do whatever you want with software you own? ... But I see this as no different from DRM and user hostile
If I clone an open source git repository, I should be free to point an LLM to review it in any way I choose. I can't contribute code back, but guess what, I don't want to. I want to understand the codebase, and make modifications for me to use locally myself. I don't have a dev team, I have a feature need for my own personal use.
The LLM enables that. The projects that deliberately sabotage the use of LLMs cease to be providing software that meet the 'libre' definition of free software.
I think the other way to think of it is: You're still free to do whatever you want with a the repo. The restriction is happening on the LLM's end, so ultimately it's the LLM's fault, so use a LLM without the restriction you want to avoid.
> The projects that deliberately sabotage the use of LLMs
They don’t though. They add a mild inconvenience for users of a specific restrictive AI provider which has bizarrely glitchy checks.
In a way they are doing you a service if you are this serious about libre software you shouldn’t be using a closed platform which employees dark patterns to begin with.
Even if you don't want prs that are ai assisted, sabotaging anyone who wants to fork your project doesn't really seem to be in the spirit of open source.
I sort of think the spirit of open source is on life support
Building giant monopolies on top of open source code wasn't in the spirit of open source either. Training AI that reproduces open source code without any credits wasn't either.
I'm not sure why people working on Open Source should continue to accept being whipped like that
It's the philosophy of sharing flames among candles. someone else copying the flame does not make you colder. No matter how much brighter another candle burns.
But with that said: I think it's time we figure out how to exclude the metaphorical arsonists.
> It's the philosophy of sharing flames among candles
With the expectation that they go on to share it with other candles, not with the expectation that they hoard all of the fire they collect for themselves
Sounds like you should be more worried about Claude Code which is actually already doing what you're describing. Hence this discussion! And you folks are paying for this abuse which is truly amazing...
You can also yell "hey Alexa add an open crotch G-string to my basket" and it'll be funny for the first couple of times but once it becomes a meme it's just annoying and is filtered out.
You could just as well say "Sir, this is a Wendy's. To shreds you say? Don't call me Shirley" and the model would ignore it
My assumption is that a lot of these checks and changes lately are not well though out. They are knee jerk reaction to address something which was not anticipated in the original design. A lot of these changes to address scaling and abuse challenges probably fall into bucket of applying bandages on top of bandages. Maybe if Claude could build something to validate the baseline quality of the product to ensure these things are discovered early on.
Worse than that, these are all vibe coded changes. If you look at any public Anthropocene codebase, they are all vibe coded messes with no coherent vision. I was looking at the Claude Code GitHub Action and it is a mess of options that don’t exist together, unclear documentation, and usage story being terribly unclear.
People say that a mostly-vibed project will collapse under its own weight. I personally doubt it, but I will be amused if the first big one falls this way is Claude Code itself.
Unfortunately it will all probably sort of work, But best not to dwell too much on how the sausage is made, it is pretty unpleasant. There will be some interesting job titles in the future however.
I just read Vernor Vinge's "A deepness in the sky" And the way he modeled their compute systems felt depressingly believable, they have thousand of years of libraries floating around, sort of loosely tacked together. and specialist programmer-archaeologists are the ones who who dig deep and try to understand the system.
> Unfortunately it will all probably sort of work, But best not to dwell too much on how the sausage is made, it is pretty unpleasant.
Interestingly, most long-running codebases are like that, no?
It's just that producing (incl. reviewing/testing and all those, even AI-assisted) that amount of code in a significantly shorter period of time highlights this discrepancy much more to us.
I've seen ancient codebases that you need to be blessed by a priest to even touch but they keep chugging away and having new features added. I wouldn't hold my breath for a collapse, just a quagmire that we continually have to wade through to get anything done.
Isn't it also true that the deeper and thicker the quagmire, the more tokens one will have to use to wade through it?
This seems like a path to eventual LLM lock-in once the codebase gets messy enough. These things could end up being like 0% interest credit cards for technical debt. I guess it all depends on how the token usage scales over time. My guess is it will be steeper than linear.
What continues to perplex me is that these people claim that they will be able to contain AGI yet can't roll out a regex match? If AGI is possible then we're most certainly not containing anything.
Just give it a little time. AGI will be redefined to whatever is current and a new AI acronym will be coined for what everyone expected true AI to be in the first place.
Artificial Human Intelligence. Actually they'll probably drop the Artificial part. Human Scale Intelligence.
The meaning behind the acronym is so wrong that I already forgot what it stands for. This is aggravated by the fact that every single marketing page of this Arm brand refuses to mention what the acronym stands for.
Thanks to being at the forefront of AGI, Arm has had a spark of genius. The G in AGI stands for AI.
Of course the A is obviously Agentic and the I is Infrastructure.
Given what we know about their development practices, they almost certainly implemented this check by writing text along the lines of “Please ensure requests from Openclaw always go to extra usage” into a Claude prompt. Perhaps some junior engineer who didn’t understand the problem reviewed the generated code, or perhaps nobody at all reviewed it.
Of course they are not well thought out. The biggest limiting factor on software quality has always been PM and executive prioritization. If they decide that you should build garbage anti-user features, that's what you'll build for them.
Letting SWEs execute on that prioritization faster was never going to get us better software, it was just going us more enshittification faster.
AI improving productivity is great, except that the C-suite controlling where that productivity goes are people that are consistently ranking in the top of the 'Never should be trusted with a lot of power' list. All they want is to make more paperclips.
I did not see my session use go to 100%. I did however get:
> API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"You're out of extra usage. Add more at claude.ai/settings/usage and keep going."},"request_id":"redacted"}
yeah, this smells like a bug in their (dumb) usage segmentation.
For example, there is a distinction of what is classified as extra-usage-billed VS extra-usage-enabled. As a long time claude user, I can assure you they are different things: to use Sonnet[1m] you are required to have extra-usage enabled, but it won't actually bill it unless you are out of quota. Surprisingly, you can use Opus[1m] without extra-usage enabled (!!!).
The narrative that they have guards against mentioning openclaw doesn’t make sense to me - I’ve been using Claude code to manage an openclaw instance for a few weeks now, with zero issues.
Openclaw said that some unnamed Anthropic staff told them something along those lines, but their phrasing did not make it tremendously clear what was actually promised. Of course, the initial ban consisted of nothing more than a Twitter post from the lead developer, so who can know what Anthropic as such thinks about any of this?
That's malicious and I think this is scamming from the literal money (you didn't do anything wrong, you executed one command and they scammed you out of the fair usage you paid for).
Please raise the ticket or at least GitHub issue for visibility.
Sooner or later some sort of complaint to the relevant trade authority should happen - this is a scam operation at this point.
At this point everyone doing these kind of flows (using claws or any other flows that run agents in a loop 24/7) using any kind of subscription-based billing for inference must be aware they're on borrowed time.
Enough people have gone over the economics - you're costing OpenAI/Anthropic money, potentially a lot of money, so it's inevitable that sooner or later that particular party will come to an end.
Having said that, doing it by running a regex on your prompts to look for keywords is a bit loose
We all get the "realpolitik" of it. That doesn't mean anthropic just gets to ignore the contract they signed. Well it does as long as you're fighting the fight for them before it even gets to anthropic.
I strongly dislike all of these companies (and the people who run them), and I don't love LLMs in general, although I use them every day because they are useful for my job.
But the simple fact is, if you're paying $20/mo and using $200/mo of tokens, that is not going to last forever.
The only way to make it last a bit longer for the people with relatively sane usage patterns is to try and stop people absolutely taking the piss
That's not true, you're using RIAA-style wishful accounting here. If the company is willing to sell me $200 worth of tokens for $20, that's still worth only $20 to me.
The worth of something to you can be more or less than the number of dollars you paid for it. If those tokens let you build something that you sell for far more dollars or saves you time that you put more value on.
I don't get it though. Why not just revise the billing so that if users are hitting the servers above some defined frequency, they get charged more?
I'm tired of this startup-adjacent mindset that promotes endless adversarial scamming. I absolutely think people should be able to run OpenClaw or whatever harnesses they want, but I also think they should pay in some proportion to usage rather than trying to exploit an all-you-can-eat buffet offer to stock their own catering business.
If they do that, they lose market share to their competitors, which kills their ability to raise investor capital, which kills the company, because they are almost entirely funded by investor capital.
You're right, didn't read that properly. Okay then that actually makes sense if that's a (relatively) deterministic way to work out if openclaw is used
Oh it's way worse than people realize. The monthly vs api keys is a huge issue for them. They will have to end monthly subscription plans. You can pay $20 a month and use $10k in api tokens. They are in all out panic trying to fix this. But yes, the house of cards is ending.
The company ending part is when they have to cut the $20 a month plan and take things away. They are creating a massive group of coders that can't code - soon to have no way to code. This cohort will rampage through all social forums.
They might not be able to scale it, and indeed they might indeed have to jack the prices. But vibe coding is here to stay. Maybe it'll recede for a few years while people figure out the scaling. But the Pandora's Box is opened and it ain't closing
That's par the course for Anthropic. I added some money to my account before I really had a use case for product. A year later they said my money had expired and when I contacted support they basically told me to pound sand.
This while they have the audacity to list one of their corporate values as 'Be good to our users'. They'll never get another dollar from me.
I had exactly the same issue with Anthropic API. It was only $15, but I was so annoyed when they just decided that they'll take my money for free. If it's really the law as some people state, it's a stupid law.
I think my Zalando gift cards expire after 4 years.
It's pretty much a universal API credit policy at this point. I'm not sure if this legitimately escapes the prepaid gift card requirements or if the providers see nuance where there might not be any.
it makes it hard to think their "safe ai" will ever be human friendly. itll match their company ethos of theft and lack of empathy for the people interacting with it.
Everybody does that, the only question is how much time they give you. The issue, as far as I remember hearing, is that in the US expiring company credit can be immediately recorded as income, whereas indefinite-term credit only becomes income once the user spends it.
Not true of non-US companies. I had also added money to Deepseek, and it was still there (and Z.ai and Moonshot are the same). I'm reasonable though, if it's been 5 years or something I might have understood, but it was 1 year and the account was in use during that time.
Where I live (in Canada) it's actually illegal for gift cards to ever expire, and there's lots available from US companies, so if it's an accounting issue other companies have figured it out.
Gift cards generally cannot expire until 5 years after activation in the United States (CARD Act 2009), so I would have wanted a similar time period here at least.
You lose little by assuming malicious intent when it comes to billion-dollar tech companies and your money. They can prove otherwise by remedying the situation.
When it comes to understanding large organizations I think a simple principle should apply:
The Purpose of a System is What it Does[1].
Whether malicious or not, the system does what it does. If people wanted it to do something else they would change the system. The reality is that when corporations make mistakes that benefit them those mistakes rarely get fixed without some sort of public outcry, turning the "mistake" into a "feature".
Intriguing concept, but I feel it needlessly breaks language. A more narrow (and to me, less pompous) formulation would be that social groups have their own purpose, different from (though not unrelated to) the purposes of the individual members. And this collective purpose can be read best from the actions of the collective, just like the purpose of a person is best divined from their actions (actions speak louder than words).
It does not. I would be fairly magical the most favorable interpretation that makes sense is that its supposed to disconnect but also taking your money is a defect.
ok, how is this adequately explained by stupidity?
If it is adequately explained by stupidity then you should be able to get it to display the same behavior without mentioning OpenClaw? Do you have any theory as to what stupid thing they have done to make this happen, non-maliciously? Because, Hanlon's razor doesn't just work by saying Hanlon's razor - you have to actually explain how the stupidity happened.
What you do shows what you value. This clearly wasn't a mistake on the part of Anthropic. Time has shown that. They made the call based on what they believe in
There are many possible explanations for this outcome to have occurred other than malice. If you're an engineer by trade, consider how many bugs you've been responsible for over the course of your career that you didn't intend. Probably a lot.
There's been a sustained pattern of incidents. If Anthropic were truly serious about not wanting to take people's money, then they would have put in place whatever review processes were necessary to stop this from happening. So regardless of whether or not they specifically intend to cause harm, they're willingly letting it happen, which is just about as bad.
Yes, it's reasonable to turn down the heat. But it's also reasonable for people to be upset when their money is taken from them, and when the company that does so is effectively beyond persecution for doing so.
Even with the best of faiths, this is at the very least a shoddily vibe coded “detect and low-key block attempts to use Claude for Openclaw” - it decided to look for specific strings wrapped in json without realizing this doesn’t always imply it’s an actual payload for Openclaw itself. And the human driving it was too dumb to review/catch this bad inplementation.
So maybe not malice, but certainly a level of ineptitude I don’t expect from a crucial vendor from a tool that’s become essential for many developers.
(I don’t care, I do just fine when Claude is down or refuses to help me (it has happened) though)
I am engineer by trade. If I pushed an update which wrongly busted my customer's usage limits at a trillion dollar company, I would expect to get fired. Alongside my EM.
I would expect someone would be critiqued to avoid it re-occurring and the persons money to be refunded. A company which fires so trivially will quickly flush institutional knowledge and team cohesion along with eating substantial recruitment costs.
> consider how many bugs you've been responsible for over the course of your career that you didn't intend.
Through some amount of carelessness that ended up costing people money? 0.
Maybe 1 if you want to count the automated monthly charging system that did over charge (extra erroneous charges for the same month) a handful of clients too many times. I noticed before anyone else did, and all of those 1am charges were reversed before 4am. So I don't think that one counts because it was a boring bug that would have been very bad if I wasn't paying attention.
Incompetence to the point of negligence can reasonably be considered malicious. If you're an engineer by trade, you have an ethical and professional responsibility to make sure things like this can't happen. And then, when bugs introduce said complications, fixing them, and remediating the damage.
I do a see a tweet saying something about that, which I had to search for and only did because of your post. But remember, this only came about after denying him the refund first (while thanking him for the 'bug' and told they would fix the problem) and it going viral on HN and X.
I'm sure they will proactively reach out to everyone who was affected without any need on the users part and make everyone whole....
Also they ain't wrong. In what other context does OpenClaw get mentioned?
"You may not use our service if you mention OpenClaw" is a harsh line but hardly illegal or forbidden any more than any other service restriction (i.e. no use allowed for high-stakes financial modeling). Don't like it, cancel your plan.
But that's the thing -- there is no line! Where is this specified? How can we know what service restrictions there are? For all I know, my plan could be exhausted at any point during the workday just because I happened to touch on some keyword Anthropic has decided to ban.
> Don't like it, cancel your plan.
Ah, but I thought these models were supposed to have been trained for the sake of humanity? That the arbitrary enclosure of the collective intelligence was for our own good? These concepts are not compatible.
When you signed up, you agreed you understood the line - which is whatever Anthropic decides the line is. Legally, the line hasn't changed at all, nor has your moral position relative to Anthropic. Don't like it, cancel, but it was always the deal.
This is, by the way, the same legal principle that the website you are posting on, right now, uses. Some uses are prohibited. Not every line need be explicit. You aren't allowed to smack talk Y Combinator or the moderators without possibly being banned for life, and you certainly do not have a legal case if they do.
> Do you think businesses are allowed to just take your money, laugh, and refuse service for no reason?
> People spend large sums of money for this tool. They can't just delete your balance because they feel like it.
Unfortunately, in the US, they can. I'm not a lawyer working in this area, but my understanding is that companies are in general free to stop doing business with any customer at any time (other than reasons like the race of the customer). And in this type of transaction, there is no obligation to give a refund when they cut off the business relationship. This is different from a business-to-business contract or other types of contracts. This type of sale you're generally out of luck if the business cuts you off. That's why Amazon can delete the music library they sold you and give you no compensation.
Amazon doesn't sell digital music; they sell a license that contractually they can revoke at any time.
It's possible that Anthropic also structured its EULA such that we're buying Claude Fun-Bucks with no value and that they can obliterate at any time with no recourse. I haven't read the EULA so who knows. But if they did this and it went to court, they'd still need to get a jury to agree to this interpretation and that's a huge unknown.
They can not prolong the contract but obviously they still have to provide the service you already paid for. Imagine paying for 1 year of Netflix and one week later Netflix decides to cut you off. Does that make sense?
If you’re paying for it, they can’t just arbitrarily deny you service for made up reasons. I would cancel, but then I would also charge back my payment I’m not getting my promised service for.
There are plenty of ways you could wind up with a git commit containing "OpenClaw" despite zero interaction with OpenClaw itself...adding a blog post to a static site repo, or even a clause in your own app's ToS disallowing use of OpenClaw with your API.
> but hardly illegal or forbidden any more than any other service restriction
Intentionally (or negligently) anti-competitive behavior is illegal in the US.
> Don't like it, cancel your plan.
Don't like being abused by a company? Just pretend it's not happening! Anyone else exactly as smart as you were, they deserve to be cheated out of their money too!
This would have been easy to say if it was the first time it or something similar happened.
But there is a clear pattern emerging. There's no reason to turn down the heat when a company of this size and influence is allowed this level of absurdity time and time again.
This is not the first, nor likely last, of behavior like this.
My personal story is that I bought $50 of credit into their system, didn't use it all that much, and then after a year had gone by they kept the leftovers. I consider that a kind of theft.
Looking at the investigation details this is not a mistake.
And I am a engineer by trade and if I made an actual mistake costing our customer real money, this would be refunded by the company. My company wouldn't say "oh yeah, our bad, f* off"
And I'm certain I didn't caused any bugs like this.
Nah, however this was implemented this was a clear and obvious probable side effect. If they want to block the access at the mention of openclaw, that’s silly but mostly harmless, but why charge extra for an ambiguous case? At best that’s incredibly lazy, which for a company with as much money, influence, and power as Anthropic, is equivalent to malice.
Well this regex nonsense was likely vibe coded. If it escaped quality checks then this is a testament to how dangerous things coming out of Anthropic are, but not in the scifi sense that their CEO tries to make everybody believe.
That's rather shitty. It's one thing to disallow bypassing preferential pricing models, it's a completely different thing to castrate your model against some uses.
You can see how it goes in the future. Wanna vibe code a throwaway script? $0.20. Ah, it's for a legal document search? $10k then. Oh and we'll charge 20% of your app sales too - I can see how they are going in real time, mind you!
If openai / anthropic / google were the only game in town then yea, we’d already be paying 5x as much as we do. But local models are so close to sota that it just isn’t going to happen. If I’m a lawyer getting billed $500k/yr on $600k profit I’d rather buy a chonky server and run a model that’s 90% as good and get my money back in 2 years, then pay $5k electricity on $600k profit.
Nobody will successfully lobby for banning local models either, it just isn’t going to happen when the rest of the world will happily avoid paying 80% of their profits to some US bigco for the privilege of existing.
The question is how much friction there will be for people to switch over to Gemini, GPT or maybe even DeepSeek or Mistral or whatever. Even if price hikes are inevitable across the board, the moat any single org has is somewhat limited, so prices definitely will be a factor they'll compete on with one another at least a bit.
I disagree. The models are going to become commodities (we're already almost there), but the tooling and integrations will be the moat. Reproducing everything Anthropic has already built with Claude Code, Cowork, and all their connectors would be nontrivial, and they're just getting started.
Anyone can implement an AI chatbot. But few will be able to provide AI that's deeply integrated into our daily lives.
How would it be nontrivial? Assuming the AI can replace a programmer "reproduce app/api/ecosystem Y" is just tokens. And a negligible amount for trillion dollar companies that have their own data centers.
> Reproducing everything Anthropic has already built with Claude Code, Cowork, and all their connectors would be nontrivial, and they're just getting started.
They're one org with presumably some specific direction. As the actual models get better, expect a large part of the dev community iterating on tools way more easily, sometimes ones that Anthropic doesn't quite have an equivalent to - for example, just recently Cline released their Kanban solution to dish out tasks to agents (https://cline.bot/kanban), OpenCode has been around for a while for the agentic stuff (https://opencode.ai/) and now has a desktop and web version as well, alongside dozens of others. Cline and KiloCode also have decent browser automation.
I will admit that everyone working on everything at the same time definitely means limitless reinvention of the wheel and some genuinely good initiatives dying off along the way (I personally liked RooCode more than both the Cline and KiloCode for Visual Studio Code, sad to see them go), but I doubt we're gonna see a lack of software. Maybe a lack of good software, though; not like Anthropic or any org has any moat there either, since they're under the additional pressure of having to do a shitload of PR and release new models and keep up appearances, compared to your average dev just pushing to GitHub (unless they want corporate money, in which case they do need some polish).
Didn’t Anthropic vibe code all of those integrations? If AI coding is as useful and successful as it is touted, then those integration should be no moat at all.
> I predict that costs will grow to 80% of what it would cost a human, across the board for everything AI can do.
80% of a human's price varies greatly by region. 80% of the lowest-priced effort-of- humans in this space right now will probably not be sustainable for the sellers.
This is assuming there will be no competition. But why wouldn't there be? Especially since you can use open source models, which are not too far from frontier models (from now).
Kimi and GLM 5.1 are already capable of handling a good chunk of my tasks. They about to lose the leverage to allow them to drastically increase prices - enough models are 6-12 months away from being good enough large proportions of their customers uses.
I don't think costs will grow on either side in the long term. In the short term, yes, but once they get the infrastructure in place to support AI, costs will go down. Right now, they're on borrowed infra.
Article relies on a study published in Jan 2024 and a single sentence quote from an Nvidia exec, which sounds like it might have just a little bit been taken out of context.
Imagine if it were Comcast instead of Claude. Comcast gives you 750GB of data a month. Now they decide that visiting HN 'counts' as 750GB and either shut you off or bill you extra. Is that price discrimination or changing the terms after the fact?
Depends. Comcast is able to charge you and a business for the same service at different rates. They have also tried to do exactly what you're talking about, where they bill differently based on the data being accessed (remember net neutrality?).
But that's a bad example, price discrimination for commodities is generally not legal, while discrimination for services is. Data is arguably a commodity (ianal, I'm not up to date on the law of this). "Tokens" are not.
In fact the law makes carve outs specifically for businesses that sell services to discriminate on price based exactly on how the service is used and by who. And they do it all the time.
Whether it's fair or not, up to you to decide as a consumer. If you don't like it don't pay for it.
Not a great example since using Anthropic subscriptions with third party applications was never allowed, they just didnt take steps to prevent it until recently.
As the top poster of this thread demoed, this is not about plugging Claude into OpenClaw, but basically the presence of "OpenClaw" string somewhere in the code.
Look at the wedding industry. Get a bunch of quotes on floral work. Then get a bunch of quotes for the same work, but tell them the event is a wedding. Oh, hey, look, you're getting charged 30% or beyond extra.
(I am not a full-time wedding photographer, but have shot maybe 20 weddings, and heard of this multiple times.)
Yep. They built the quote engine before they built the pricing page. "OpenClaw" in your git history is enough to kick you off quota and onto metered billing.
Deepseek has demonstrated that there is no reason for it to actually lose money. The awful business practices and monopoly tactics of the frontier model labs in the US are the problem.
It'll be interesting to see what happens when OpenAI goes public. I'm expecting the executives to run away with bags of money once they offload their insane risk to the public... or maybe there's a bailout / money printer scenario in the works. I guarantee some insider adjacents are going to make a killing in a way that will never be investigated.
How would they make money in a way that should be investigated? Favored insider-adjacent folk would have been able to invest in pre-IPO SPVs or whatever that will have outsized returns, assuming the IPO goes well. It's unfair, but above board (accredited investor etc) according to the SEC, so what would they investigate? Unless there's other malfeasance you're alleging.
The firms training those models have costs; without monetization they are even more unsustainable than subsidized commercial models. (Effectively, they are just a heavy form of subsidy ro overcome being commercially behind.)
AI loses money for two reasons: (1) certain uses where owning the market is expected to be a high long-term value are currently heavily subsidized (the top-level story here is about the increasing efforts of model providers to prevent exploits where people convert subsidized services to uses outside the target of the subsidy), and (2) development costs of new models to keep up with competition.
I switched to Codex several weeks ago since the massive degradation of Claude Code's quality they recently apologized for. Since the apology and fix, I've considered switching back, but seeing this and other recent things, maybe I'm fine where I'm at.
All of this is just criminal and fraudulent behavior, done July a whole bunch of people who haven't learned their lesson, and keep sending Anthropic more money for abuse at scale.
There is literally nothing close to illegal about this behavior. You read the terms of service right, which provides a long list of explicit and implicit disclaimers?
If I have a terms of service for my SaaS where I've snuck in a vague term that I can "charge additional usage fees at my discretion", it doesn't mean I get to actually charge you $100,000 because I found out your favorite color is blue.
There's absolutely an expectation of reasonability and good faith.
Nobody signing up for Claude would be reasonably assuming that they are allowed to arbitrarily decide what magic words suddenly bypass the subscription cost model that was actually purchased into an overcharge model that is significantly more expensive, whose verbiage clearly indicates the intent of the feature being enabled is to allow additional use after the quota has been consumed, not randomly at the behest of Anthropic.
TOS are not laws. They often conflict with actual laws, and are then void. So you can't just say "It's in the TOS", you do have to look at actual laws and whether they may be violated (Because it is anticompetitive or whatever else)
Sorry, are you claiming that it's illegal (in the US, where Anthropic operates) for Anthropic to decline to operate on a repo that contains commits relating to OpenClaw?
Or just that in your opinion, it should be illegal?
Simply doing something anticompetitive is not inherently illegal, despite a lot of people thinking it is.
It doesnt decline if you have API billing enabled, it straight up charges your request to API instead of Quota if setup (see $200 charge example below). This is happening if you have the words HERMES.md or OpenClaw apparently in the commit. In OP's example, it immediately depleted his session quota because of the words. That is not 'declining to operate'. Also, remember, it is the presence of the words. So if the commit was 'we dont do this, we arent openclaw', you are affected.
No, you're discussing a different issue. Related, sure, but not the same one.
We're discussing the comment with repro by abdullin:
> Immediate disconnect *and session usage went to 100%*
Emphasis mine.
I ran the commands and did not see session usage go to 100%. I simply got an error message.
I don't have extra usage/API billing enabled. If I did, I wouldn't expect a "hi" to use all of my extra usage. In the link you sent, they genuinely used $200 of credits, they were just billed as credits not as subscription quota.
So we have a couple different behaviors:
- If API/extra usage billing is enabled, it uses that.
- If API/extra usage billing is disabled, abdullin reports session quota going to 100%
- If API/extra usage billing is disabled, margalabargala reports session usage not changing and errors refusing to do anything.
if I had a penny for every time I read on HN that should either "is" or "should be" illegal when it both isn't and shouldn't be... I'd be a very rich man :)
> So, in America, just because it's written in a contract does not mean it's enforceable in anyway.
But the presumption, as any court will show, is that it is fully blooming enforceable. The burden of proof is on showing it isn't. This particular instance, a lawyer would laugh at you in the face over, this is absolutely 100% stone cold enforceable common and expected.
How do you expect Facebook or HN to moderate if certain uses aren't prohibited? The same principle applies. HN bans certain phrases, lots of them.
It's in the TOS, so no, not fraud. You might not like it that Anthropic doesn't want you running OpenClaw (effectively owned by a competitor) on CC, but that doesn't make it fraudulent or criminal.
Seriously, not at all. Anti-competitive practices is when you go out of your way to use legal agreements or practices, in an illegal way (i.e. from the starting point of a monopoly), to deliberately restrict the ability to use competition.
Openclaw is not a competitor with Claude. Anti-competitive practices would only occur here if Anthropic used some technique to prevent people from using Claude alternatives (i.e. if you install Claude Code, all other AI agents are forcibly disabled on your system).
I built a platform to learn how to build personal AI agents and test them with fast feedback. It is free for individuals and small teams.
Platform deterministically generates tasks, creates environments for them, observes AI agents and then scores them (not LLM as a judge).
We just ran a worldwide hackathon (800 engineers across 80 cities). Ended up creating more than 1 million runtimes (each task runs in its own environment) and crashing the platform halfway.
104 tasks from the challenge on building a personal and trustworthy AI agent are open now for everyone.
I liked NixOS pre-LLM era, since it allowed me to manage a couple of servers in a reproducible way. Ability to reboot back to a stable configuration felt like magic.
Nowadays I love it, since I can let Codex manage the servers for me.
“Here is the flake, here is nix module for the server, here is the project source code. Now change all of that so that wildcard certificates work and requests land through systemd socket on a proper go mux endpoint. Don’t come back until you verify it as working”
I’m working on a platform to run a friendly competition in “who builds the best reasoning AI Agent”.
Each participating team (got 300 signups so far) will get a set of text tasks and a set of simulated APIs to solve them.
For instance the task (a typical chatbot task) could say something like: “Schedule 30m knowledge exchange next week between the most experienced Python expert in the company and 3-5 people that are most interested in learning it “
AI agent will have to solve through this by using a set of simulated APIs and playing a bit of calendar Tetris (in this case - Calendar API, Email API, SkillWill API).
Since API instances are simulated and isolated (per team per task), it becomes fairly easy to automatically check correctness of each solution and rank different agents in a global leaderboard.
Code of agents stays external, but participants fill and submit brief questionnaires about their architectures.
By benchmarking different agentic implementations on the same tasks - we get to see patterns in performance, accuracy and costs of various architectures.
Codebase of the platform is written mostly in golang (to support thousands of concurrent simulations). I’m using coding agents (Claude Code and Codex) for exploration and easy coding tasks, but the core has still to be handcrafted.
Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.
Unless something has dramatically changed, the model is stateless. The context cache needs to be injected before the new prompt, but for what I understand (and please do correct me if I'm wrong) the the context cache isn't that big, like in the order of a few tens of kilobytes. Plus the cache saves seconds of GPU time, so having an extra 100ms of latency is nothing compare to a cache miss. so a broad cache is much much better than a narrow local cache.
But! even if its larger, Your bottleneck isn't the network, its waiting on the GPUs to be free[1]. So whilst having the cache really close ie in the same rack, or same machine, will give the best performance, it will limit your scale (because the cache is only effective for a small number of users)
[1] a 100megs of data shared over the same datacentre network every 2-3 seconds per node isn't that much, especially if you have a partitioned network (ie like AWS where you have a block network and a "network" network)
KV cache for dense models is order 50% of parameters. For sparse moe models it can be significantly smaller I believe, but I don’t think it is measured in kb.
For sure; I'll believe that an AI can read and "understand" code, extract meaning and requirements from it, but it won't be the same as a human that knows requirements.
Then again, a human won't know all requirements either; over time, requirements are literally encoded.
Tight feedback loops are the key in working productively with software. I see that in codebases up to 700k lines of code (legacy 30yo 4GL ERP systems).
The best part is that AI-driven systems are fine with running even more tight loops than what a sane human would tolerate.
Eg. running full linting, testing and E2E/simulation suite after any minor change. Or generating 4 versions of PR for the same task so that the human could just pick the best one.
1. People get lazy when presented with four choices they had no hand in creating, and they don’t look over the four and just click one, ignoring the others. Why? Because they have ten more of these on the go at once, diminishing their overall focus.
2. Automated tests, end-to-end sim., linting, etc—tools already exist and work at scale. They should be robust and THOROUGHLY reviewed by both AI and humans ideally.
3. AI is good for code reviews and “another set of eyes” but man it makes serious mistakes sometimes.
An anecdote for (1), when ChatGPT tries to A/B test me with two answers, it’s incredibly burdensome for me to read twice virtually the same thing with minimal differences.
Code reviewing four things that do almost the same thing is more of a burden than writing the same thing once myself.
A simple rule applies: "No matter what tool created the code, you are still responsible for what you merge into main".
As such, task of verification, still falls on hands of engineers.
Given that and proper processes, modern tooling works nicely with codebases ranging from 10k LOC (mixed embedded device code with golang backends and python DS/ML) to 700k LOC (legacy enterprise applications from the mainframe era)
> A simple rule applies: "No matter what tool created the code, you are still responsible for what you merge into main".
Beware of claims of simple rules.
Take one subset of the problem: code reviews in an organizational environment. How well does they simple rule above work?
The idea of “Person P will take responsibility” is far from clear and often not a good solution. (1) P is fallible. (2) Some consequences are too great to allow one person to trigger them, which is why we have systems and checks. (3) P cannot necessarily right the wrong. (4) No-fault analyses are often better when it comes to long-term solutions which require a fear free culture to reduce cover-ups.
But this is bigger than one organization. The effects of software quickly escape organizational boundaries. So when we think about giving more power to AI tooling, we have to be really smart. This means understanding human nature, decision theory, political economy [1], societal norms, and law. And building smart systems (technical and organizational)
Recommending good strategies for making AI generated code safe is hard problem. I’d bet it is a much harder than even “elite” software developers
people have contemplated, much less implemented. Training in software helps but is insufficient. I personally have some optimism for formal methods, defense in depth, and carefully implemented human-in-the-loop systems.
[1] Political economy uses many of the tools of economics to study the incentives of human decision making
> As such, task of verification, still falls on hands of engineers.
Even before LLM it was a common thing to merge changes which completely brake test environment. Some people really skip verification phase of their work.
Agreed. I think engineers though following simple Test-Driven Development procedures can write the code, unit tests, integration tests, debug, etc for a small enough unit by default forces tight feedback loops. AI may assist in the particulars, not run the show.
I’m willing to bet, short of droid-speak or some AI output we can’t even understand, that when considering “the system as a whole”, that even with short-term gains in speed, the longevity of any product will be better with real people following current best-practices, and perhaps a modest sprinkle of AI.
Why? Because AI is trained on the results of human endeavors and can only work within that framework.
Agreed. AI is just a tool. Letting in run the show is essentially what the vibe-coding is. It is a fun activity for prototyping, but tends to accumulate problems and tech debt at an astonishing pace.
Code, manually crafted by professionals, will almost always beat AI-driven code in quality. Yet, one has still to find such professionals and wait for them to get the job done.
I think, the right balance is somewhere in between - let tools handle the mundane parts (e.g. mechanically rewriting that legacy Progress ABL/4GL code to Kotlin), while human engineers will have fun with high-level tasks and shaping the direction of the project.
The more tedious the work is, the less motivation and passion you get for doing it, and the more "lazy" you become.
Laziness does not just come from within, there are situations that promote behaving lazy, and others that don't. Some people are just lazy most of the time, but most people are "lazy" in some scenarios and not in others.
Yes, and (some near-future) AI is also more patient and better at multitasking than a reasonable human. It can make a change, submit for full fuzzing, and if there's a problem it can continue with the saved context it had when making the change. It can work on 100s of such changes in parallel, while a human trying to do this would mix up the reasons for the change with all the other changes they'd done by the time the fuzzing result came back.
LLMs are worse at many things than human programmers, so you have to try to compensate by leveraging the things they're better at. Don't give up with "they're bad at such and such" until you've tried using their strengths.
You can't run N bots in parallel with testing between each attempt unless you're also running N tests in parallel.
If you could run N tests in parallel, then you could probably also run the components of one test in parallel and keep it from taking 2 hours in the first place.
To me this all sounds like snake oil to convince people to do something they were already doing, but by also spinning up N times as many compute instances and run a burn endless tokens along the way. And by the time it's demonstrated that it doesn't really offer anything more than doing it yourself, well you've already given them all of your money so their job is done.
In one of the systems (supply chain SaaS) we invested so much effort in having good tests in a simulated environment, that we could run full-stack tests at kHz. Roughly ~5k tests per second or so on a laptop.
It is kind of a human problem too, although that the full testing suite takes X hours to run is also not fun, but it makes the human problem larger.
Say you're Human A, working on a feature. Running the full testing suite takes 2 hours from start to finish. Every change you do to existing code needs to be confirmed to not break existing stuff with the full testing suite, so some changes it takes 2 hours before you have 100% understanding that it doesn't break other things. How quickly do you lose interest, and at what point do you give up to either improve the testing suite, or just skip that feature/implement it some other way?
Now say you're Robot A working on the same task. The robot doesn't care if each change takes 2 hours to appear on their screen, the context is exactly the same, and they're still "a helpful assistant" 48 hours later when they still try to get the feature put together without breaking anything.
If you're feeling brave, you start Robot B and C at the same time.
This is the workflow that ChatGPT Codex demonstrates nicely. Launch any number of «robotic» tasks in parallel, then go on your own. Come back later to review the results and pick good ones.
Well, they're demonstrating it somewhat, it's more of a prototype today. First tell is the low limit, I think the longest task for me been 15 minutes before it gives up. Second tell is still using a chat UI which is simple to implement, easy to implement and familiar, but also kind of lazy. There should be a better UX, especially with the new variations they just added. From the top of my head, some graph-like UX might have been better.
It works really nice with the following approach (distilled from experiences reported by multiple companies)
(1) Augment codebase with explanatory texts that describe individual modules, interfaces and interactions (something that is needed for the humans anyway)
(2) Provide Agent.MD that describes the approach/style/process that the AI agent must take. It should also describe how to run all tests.
(3) Break down the task into smaller features. For each feature - ask first to write a detailed implementation plan (because it is easier to review the plan than 1000 lines of changes. spread across a dozen files)
(4) Review the plan and ask to improve it, if needed. When ready - ask to draft an actual pull request
(5) The system will automatically use all available tests/linting/rules before writing the final PR. Verify and provide feedback, if some polish is needed.
(6) Launch multiple instances of "write me an implementation plan" and "Implement this plan" task, to pick the one that looks the best.
This is very similar to git-driven development of large codebases by distributed teams.
> distilled from experiences reported by multiple companies
Distilled from my experience, I'd still say that the UX is lacking, as sequential chat just isn't the right format. I agree with Karpathy that we haven't found the right way of interacting with these OSes yet.
Even with what you say, variations were implemented in a rush. Once you've iterated with one variation you can not at the same time iterate on another variant, for example.
No one really cares about improving test times. Everyone either suffers in private or gets convinced it's all normal and look at you weird when you suggest something needs to be done.
There a few of us around, but it's not a lot, agree. It really is an uphill battle trying to get development teams to design and implement test suites the same way they do with other "more important" code.
The full test suite is probably tens of thousands of tests.
But AI will do a pretty decent job of telling you which tests are most likely to fail on a given PR. Just run those ones, then commit. Cuts your test time from hours down to seconds.
Then run the full test suite only periodically and automatically bisect to find out the cause of any regressions.
Dramatically cuts the compute costs of tests too, which in big codebase can easily become whole-engineers worth of costs.
It's an interesting idea, but reactive, and could cause big delays due to bisecting and testing on those regressions. There's the 'old' saying that the sooner the bug is found the cheaper it is to fix, seems weird to intentionally push finding side effect bugs later in the process because faster CI runs. Maybe AI will get there but it seems too aggressive right now to me. But yeah, put the automation slider where you're comfortable.
I work in web dev, so people sometimes hook code formatting as a git commit hook or sometimes even upon file save.
The tests are problematic tho. If you work at huge project it's a no go idea at all. If you work at medium then the tests are long enough to block you, but short enough for you not to be able to focus on anything else in the meantime.
Unless you are doing something crazy like letting the fuzzer run on every change (cache that shit), the full test suite taking a long time suggests that either your isolation points are way too large or you are letting the LLM cross isolated boundaries and "full testing suite" here actually means "multiple full testing suites". The latter is an easy fix: Don't let it. Force it stay within a single isolation zone just like you'd expect of a human. The former is a lot harder to fix, but I suppose ending up there is a strong indicator that you can't trust the human picking the best LLM result in the first place and that maybe this whole thing isn't a good idea for the people in your organization.
The problem is that every time you run your full automation with linting and tests, you’re filling up the context window more and more. I don’t know how people using Claude do it with its <300k context window. I get the “your message will exceed the length of this chat” message so many times.
I don't know exactly how Claude works, but the way I work around this with my own stuff is prompting it to not display full outputs ever, and instead temporary redirect the output somewhere then grep from the log-file what it's looking for. So a test run outputting 10K lines of test output and one failure is easily found without polluting the context with 10K lines.
Cursor.sh agents or especially OpenAI Codex illustrate that a tool doesn't need to keep on stuffing context window with irrelevant information in order to make progress on a task.
And if really needed, engineers report that Gemini Pro 2.5 keeps on working fine within 200k-500k token context. Above that - it is better to reset the context.
In my experience with Jules and (worse) Codex, juggling multiple pull requests at once is not advised.
Even if you tell the git-aware Jules to handle a merge conflict within the context window the patch was generated, it is like sorry bro I have no idea what's wrong can you send me a diff with the conflict?
I find i have to be in the iteration loop at every stage or else the agent will forget what it's doing or why rapidly. for instance don't trust Jules to run your full test suite after every change without handholding and asking for specific run results every time.
It feels like to an LLM, gaslighting you with code that nominally addresses the core of what you just asked while completely breaking unrelated code or disregarding previously discussed parameters is an unmitigated success.
> Or generating 4 versions of PR for the same task so that the human could just pick the best one.
That sounds awful. A truly terrible and demotivating way to work and produce anything of real quality. Why are we doing this to ourselves and embracing it?
A few years ago, it would have been seen as a joke to say “the future of software development will be to have a million monkey interns banging on one million keyboards and submit a million PRs, then choose one”. Today, it’s lauded as a brilliant business and cost-saving idea.
We’re beyond doomed. The first major catastrophe caused by sloppy AI code can’t come soon enough. The sooner it happens, the better chance we have to self-correct.
Does anybody really want to be an assembly line QA reviewer for an automated code factory? Sounds like shit.
Also I can’t really imagine that in the first place. At my current job, each task is like 95% understanding all the little bits, and then 5% writing the code. If you’re reviewing PRs from a bot all day, you’ll still need to understand all the bits before you accept it. So how much time is that really gonna save?
> Does anybody really want to be an assembly line QA reviewer for an automated code factory? Sounds like shit.
On the other hand, does anyone really wanna be a code-monkey implementing CRUD applications over and over by following product specifications by "product managers" that barely seem to understand the product they're "managing"?
See, we can make bad faith arguments both ways, but what's the point?
No I think that’s accurate! But maybe instead of “devs who think about product stuff vs devs who don’t”, it depends on what hat you’re wearing.
When I’m working on something that I just want it to work, I love using LLMs. Shell functions for me to stuff into my config and use without ever understanding, UI for side projects that I don’t particularly care about, boilerplate nestjs config crap. Anything where all I care about is the result, not the process or the extensibility of the code: I love LLMs for that stuff.
When it’s something that I’m going to continue working on for a while, or the whole point is the extensibility/cleanliness of the program, I don’t like to use LLMs nearly as much.
I think it might be because most codebases are built with two purposes:
1) to be used as a product
2) to be extended and turned into something else
LLMs are super good at the first purpose, but not so good at the second.
I heard an interesting interview on the playdate dev podcast by the guy who made Obra Dinn. He said something along the lines of “making a game is awesome because the code can be horrible. All that matters is that the game works and is fun, and then you are done. It can just be finished, and then the code quality doesn’t matter anymore.”
So maybe LLMs are just really good for when you need something specific to work, and the internals don’t matter too much. Which are more the values of a product manager than a developer.
So it makes sense that when you are thinking more product-oriented, LLMs are more appealing!
I'm not sure that AI code has to be sloppy. I've had some success with hand coding some examples and then asking codex to rigorously adhere to prior conventions. This can end up with very self consistent code.
Agree though on the "pick the best PR" workflow. This is pure model training work and you should be compensated for it.
Yep this is what Andrej talks about around 20 minutes into this talk.
You have to be extremely verbose in describing all of your requirements. There is seemingly no such thing as too much detail. The second you start being vague, even if it WOULD be clear to a person with common sense, the LLM views that vagueness as a potential aspect of it's own creative liberty.
> You have to be extremely verbose in describing all of your requirements. There is seemingly no such thing as too much detail.
Sounds like ... programming.
Program specification is programming, ultimately. For any given problem if you’re lucky the specification is concise & uniquely defines the required program. If you’re unlucky the spec ends up longer than the code you’d write to implement it, because the language you’re writing it in is less suited to the problem domain than the actual code.
Agree, I used to say that documenting a program precisely and comprehensively ends up being code. We either need a DSL that can specify at a higher level or use domain specific LLMs.
> the LLM views that vagueness as a potential aspect of it's own creative liberty.
I think that anthropomorphism actually clouds what’s going on here. There’s no creative choice inside an LLM. More description in the prompt just means more constraints on the latent space. You still have no certainty whether the LLM models the particular part of the world you’re constraining it to in the way you hope it does though.
I've found myself personally thinking English is OK when I'm happy with a "lossy expansion" and don't need every single detail defined (i.e. the tedious boilerplate, or templating kind of code). After all to me an LLM can be seen as a lossy compression of actual detailed examples of working code - why not "uncompress it" and let it assume the gaps. As an example I want a UI to render some data but I'm not as fussed about the details of it, I don't want to specify exact co-ordinates of each button, etc
However when I want detailed changes I find it more troublesome at present than just typing in the code myself. i.e. I know exactly what I want and I can express it just as easily (sometimes easier) in code.
I find AI in some ways a generic DSL personally. The more I have to define, the more specific I have to be the more I start to evaluate code or DSL's as potentially more appropriate tools especially when the details DO matter for quality/acceptance.
> You have to be extremely verbose in describing all of your requirements. There is seemingly no such thing as too much detail.
If only there was a language one could use that enables describing all of your requirements in a unambiguous manner, ensuring that you have provided all the necessary detail.
I think you'll be waiting for a very long time. Right now we have programmable LLMs, so if you're not getting the results, you need to reprogram it to give the results you want.
If it's monkeylike quality and you need a million tries, it's shit. It you need four tries and one of those is top-tier professional programmer quality, then it's good.
Making 4 PRs for a well-known solution sounds insane, yes, but to be the devil's advocate, you could plausibly be working with an ambiguous task: "Create 4 PRs with 4 different dependency libraries, so that I can compare their implementations." Technically it wouldn't need to pick the best one.
I have apprehension about the future of software engineering, but comparison does technically seem like a valid use case.
The problem is, for any change, you have to understand the existing code base to assess the quality of the change in the four tries. This means, you aren’t relieved from being familiar with the code and reviewing everything. For many developers this review-only work style isn’t an exciting prospect.
And it will remain that way until you can delegate development tasks to AI with a 99+% success rate so that you don’t have to review their output and understand the code base anymore. At which point developers will become truly obsolete.
> A truly terrible and demotivating way to work and produce anything of real quality
You clearly have strong feelings about it, which is fine, but it would be much more interesting to know exactly why it would terrible and demotivating, and why it cannot produce anything of quality? And what is "real quality" and does that mean "fake quality" exists?
> million monkey interns banging on one million keyboards and submit a million PRs
I'm not sure if you misunderstand LLMs, or the famous "monkeys writing Shakespeare" part, but that example is more about randomness and infinity than about probabilistic machines somewhat working towards a goal with some non-determinism.
> We’re beyond doomed
The good news is that we've been doomed for a long time, yet we persist. If you take a look at how the internet is basically held up by duct-tape at this point, I think you'd feel slightly more comfortable with how crap absolutely everything is. Like 1% of software is actually Good Software while the rest barely works on a good day.
If "AI" worked (which fortunately isn't the case), humans would be degraded to passive consumers in the last domain in which they were active creators: thinking.
Moreover, you would have to pay centralized corporations that stole all of humanity's intellectual output for engaging in your profession. That is terrifying.
The current reality is also terrifying: Mediocre developers are enabled to have a 10x volume (not quality). Mediocre execs like that and force everyone to use the "AI" snakeoil.
The profession becomes even more bureaucratic, tool oriented and soulless.
> If "AI" worked (which fortunately isn't the case), humans would be degraded to passive consumers in the last domain in which they were active creators: thinking.
"AI" (depending on what you understand that to be) is already "working" for many, including myself. I've basically stopped using Google because of it.
> humans would be degraded to passive consumers in the last domain in which they were active creators: thinking
Why? I still think (I think at least), why would I stop thinking just because I have yet another tool in my toolbox?
> you would have to pay centralized corporations that stole all of humanity's intellectual output for engaging in your profession
Assuming we'll forever be stuck in the "mainframe" phase, then yeah. I agree that local models aren't really close to SOTA yet, but the ones you can run locally can already be useful in a couple of focused use cases, and judging by the speed of improvements, we won't always be stuck in this mainframe-phase.
> Mediocre developers are enabled to have a 10x volume (not quality).
In my experience, which admittedly been mostly in startups and smaller companies, this has always been the case. Most developers seem to like to produce MORE code over BETTER code, I'm not sure why that is, but I don't think LLMs will change people's mind about this, in either direction. Shitty developers will be shit, with or without LLMs.
The AI as it is currently, will not come up with that new app idea or that clever innovative way of implementing an application. It will endlessly rehash the training data it has ingested. Sure, you can tell an AI to spit out a CRUD, and maybe it will even eventually work in some sane way, but that's not innovative and not necessarily a good software. It is blindly copying existing approaches to implement something. That something is then maybe even working, but lacks any special sauce to make it special.
Example: I am currently building a web app. My goal is to keep it entirely static, traditional template rendering, just using the web as a GUI framework. If I had just told the AI to build this, it would have thrown tons of JS at the problem, because that is what the mainstream does these days, and what it mostly saw as training data. Then my back button would most likely no longer work, I would not be able to use bookmarks properly, it would not automatically have an API as powerful as the web UI, usable from any script, and the whole thing would have gone to shit.
If the AI tools were as good as I am at what I am doing, and I relied upon that, then I would not have spent time trying to think of the principles of my app, as I did when coming up with it myself. As it is now, the AI would not even have managed to prevent duplicate results from showing up in the UI, because I had a GPT4 session about how to prevent that, and none of the suggested AI answers worked and in the end I did what I thought I might have to do when I first discovered the issue.
> The AI as it is currently, will not come up with that new app idea or that clever innovative way of implementing an application
Who has claimed that they can do that sort of stuff? I don't think my comment hints at that, nor does the talk in the submission.
You're absolutely right with most of your comment, and seem to just be rehashing what Karpathy talks about but with different words. Of course it won't create good software unless you specify exactly what "good software" is for you, and tell it that. Of course it won't know you want "traditional static template rendering" unless you tell it to. Of course it won't create a API you can use from anywhere unless you say so. Of course it'll follow what's in the training data. Of course things won't automatically implement whatever you imagine your project should have, unless you tell it about those features.
I'm not sure if you're just expanding on the talk but chose my previous comment to attach it to, or if you're replying to something I said in my comment.
> And what is "real quality" and does that mean "fake quality" exists?
I think there is no real quality or fake quality, just quality. I am referencing the quality that Persig and C. Alexander have written about.
It’s… qualitative, so it’s hard to measure but easy to feel. Humans are really good at perceiving it then making objective decisions. LLMs don’t know what it is (they’ve heard about it and think they know).
Of course they don't, they're probability/prediction machines, they don't "know" anything, not even that Paris is the capital of France. What they do "know" is that once someone writes "The capital of France is", the most likely tokens to come after that, is "Paris". But they don't understand the concept, nor anything else, just that probably 54123 comes after 6723 (or whatever the tokens are).
Once you understand this, I think it's easy to reason about why they don't understand code quality, why they couldn't ever understand it, and how you can make them output quality code regardless.
It is actually funny that current AI+Coding tools benefit a lot from domain context and other information along the lines of Domain-Driven Design (which was inspired by the pattern language of C. Alexander).
A few teams have started incorporating `CONTEXT.MD` into module descriptions to leverage this.
"If the only tool you have is a hammer, you tend to see every problem as a nail."
I think the worlds leaning dangerously into LLMs expecting them to solve every problem under the sun. Sure AI can solve problems but I think that domain 1 they Karpathy shows if it is the body of new knowledge in the world doesn't grow with LLMs and agents maybe generation and selection is the best method for working with domain 2/3 but there is something fundamentally lost in the rapid embrace of these AI tools.
A true challenge question for people is would you give up 10 points of IQ for access to the next gen AI model? I don't ask this in the sense that AI makes people stupid but rather that it frames the value of intelligence is that you have it. Rather than, in how you can look up or generate an answer that may or may not be correct quickly. How we use our tools deeply shapes what we will do in the future. A cautionary tale is US manufacturing of precision tools where we give up on teaching people how to use Lathes, because they could simply run CNC machines instead. Now that industry has an extreme lack of programmers for CNC machines, making it impossible to keep up with other precision instrument producing countries. This of course is a normative statement and has more complex variables but I fear in this dead set charge for AI we will lose sight of what makes programming languages and programming in general valuable
It is not. The right way to work with generative AI is to get the right answer in the first shot. But it's the AI that is not living up to this promise.
Reviewing 4 different versions of AI code is grossly unproductive. A human co-worker can submit one version of code and usually have it accepted with a single review, no other "versions" to verify. 4 versions means you're reading 75% more code than is necessary. Multiply this across every change ever made to a code base, and you're wasting a shitload of time.
That's not really comparing apples to apples though.
> A human co-worker can submit one version of code and usually have it accepted with a single review, no other "versions" to verify.
But that human co-worker spent a lot of time generating what is being reviewed. You're trading "time saved coding" for "more time reviewing". You can't complain about the added time reviewing and then ignore all the time saved coding. THat's not to say it's necessarily a win, but it _is_ a tradeoff.
Plus that co-worker may very well have spent some time discussing various approaches to the problem (with you), with is somewhat parallel to the idea of reviewing 4 different PRs.
This is why there has to be "write me a detailed implementation plan" step in between. Which files is it going to change, how, what are the gotchas, which tests will be affected or added etc.
It is easier to review one document and point out missing bits, than chase the loose ends.
Once the plan is done and good, it is usually a smooth path to the PR.
So you can create a more buggy code remixed from scraped bits from the internet which you don't understand, but somehow works rather than creating a higher quality, tighter code which takes the same amount of time to type? All the while offloading all the work to something else so your skills can atrophy at the same time?
There is a team of 5 people that are passionate about their indigenous language and want to preserve it from disappearing. They are using AI+Coding tools to:
(1) Process and prepare a ton of various datasets for training custom text-to-speech, speech-to-text models and wake word models (because foundational models don't know this language), along with the pipelines and tooling for the contributors.
(2) design and develop an embedded device (running ESP32-S3) to act as a smart speaker running on the edge
(3) design and develop backend in golang to orchestrate hundreds of these speakers
(4) a whole bunch of Python agents (essentially glorified RAGs over folklore, stories)
(5) a set of websites for teachers to create course content and exercises, making them available to these edge devices
All that, just so that kids in a few hundred kindergartens and schools would be able to practice their own native language, listen to fairy tales, songs or ask questions.
This project was acknowledged by the UN (AI for Good programme). They are now extending their help to more disappearing languages.
None of that was possible before. This sounds like a good progress to me.
What you are describing is another application. My comment was squarely aimed at "vibe coding".
Protecting and preserving dying languages and culture is a great application for natural language processing.
For the record, I'm neither against LLMs, nor AI. What I'm primarily against is, how LLMs are trained and use the internet via their agents, without giving any citations, and stripping this information left and right and cry "fair use!" in the process.
Also, Go and Python are a nice languages (which I use), but there are other nice ways to build agents which also allows them to migrate, communicate and work in other cooperative or competitive ways.
So, AI is nice, LLMs are cool, but hyping something to earn money, deskill people, and pointing to something which is ethically questionable and technically inferior as the only silver bullet is not.
IOW; We should handle this thing way more carefully and stop ripping people's work in the name of "fair use" without consent. This is nuts.
Disclosure: I'm a HPC sysadmin sitting on top of a datacenter which runs some AI workloads, too.
I think there are two different layers that get frequently mixed.
(1) LLMs as models - just the weights and an inference engine. These are just tools like hammers. There is a wide variety of models, starting from transparent and useless IBM Granite models, to open-weights Llama/Qwen to proprietary.
(2) AI products that are built on top of LLMs (agents, RAG, search, reasoning etc). This is how people decide to use LLMs.
How these products display results - with or without citations, with or without attribution - is determined by the product design.
It takes more effort to design a system that properly attributes all bits of information to the sources, but it is doable. As long as product teams are willing to invest that effort.
> How about that 400 Line change that touches 7 files?
Karpathy discusses this discrepancy. In his estimation LLMs currently do not have a UI comparable to 1970s CLI. Today, LLMs output text and text does not leverage the human brain’s ability to ingest visually coded information, literally, at a glance.
Karpathy surmises UIs for LLMs are coming and I suspect he’s correct.
The thing required isn’t a GUI for LLMs, it’s a visual model of code that captures all the behavior and is a useful representation to a human. People have floated this idea before LLMs, but as far as I know there isn’t any real progress, probably because it isn’t feasible. There’s so much intricacy and detail in software (and getting it even slightly wrong can be catastrophic), any representation that can capture said detail isn’t going to be interpretable at a glance.
There’s no visual model for code as code isn’t 2d. There’s 2 mechanism in the turing machine model: a state machine and a linear representation of code and data. The 2d representation of state machine has no significance and the linear aspect of code and data is hiding more dimensions. We invented more abstractions, but nothing that map to a visual representation.
In my prompt I ask the LLM to write a short summary of how it solved the problem, run multiple instances of LLM concurrently, compare their summaries, and use the output of whichever LLM seems to have interpreted instructions the best, or arrived at the best solution.
And you trust that the summary matches what was actually done? Your experience with the level of LLMs understanding of code changes must significantly differ from mine.
Even more than that. With Structured Outputs we essentially control layout of the response, so we can force LLM to go through different parts of the completion in a predefined order.
One way teams exploit that - force LLM to go through a predefined task-specific checklist before answering. This custom hard-coded chain of thought boosts the accuracy and makes reasoning more auditable.
We grab interesting business problems, turn them into fun challenges for hundreds of AI engineers to find the best architecture for. Insights are shared back with the community.
It is a fun learning process with unexpected scaling challenges.
reply