Ever since Microsoft's acquisition of GitHub 8 years ago, GitHub has completely enshittified and has become so unreliable, that even self-hosting a Git repository or self-hosted actions yourself would have a far better uptime than GitHub.
This sounded crazy in 2020 when I said that in [0]. Now it doesn't in 2026 and many have realized how unreliable GitHub has become.
If there was a prediction market on the next time GitHub would have at least one major outage per week, you would be making a lot of money since it appears that AI chatbots such as Tay.ai, Zoe and Copilot are somewhat in charge of wrecking the platform.
Any other platform wouldn't tolerate such outages.
Also having to wait for ChatGPT for a "thinking" response to search for information that is slower than a Google search loses them lots of money.
I believe that it can still work and I won't claim about being unsurprised about this failure. But this is a great opportunity to execute this problem really well if OpenAI and others are not interested in getting good at this.
Perplexity also attempted this, got sued by Amazon and it appears semi-abandoned.
The only problem is that it must be quicker or just as quick as a Google search, and also compatible with the existing checkout flows.
> Perplexity also attempted this, got sued by Amazon and it appears semi-abandoned.
Any details on that? I feel the answer is more likely there than in "friction".
Hardly any purchase of consequence is so sensitive to friction that the difference between Google Search and an LLM response matters (especially that in reality, we're talking 20+ manual searches per one LLM response). I.e. I'm not going to use LLMs advise on some random 0-100$ purchase anyway, and losing #$ on a ##$ purchase due to suboptimal choice is not that big of a deal - but I absolutely am going to consult it (and have it compile tables and verify sources) on a $500+ purchase and for those I can afford spending few more minutes on research (or rather few hours less, compared of doing it the usual way).
This was obvious to those who value their time over the job given to them and all the office politics, performative meetings and the blame-game that comes with it.
The "tech jobs" you are looking for are actually potemkin ghost jobs that are never going to be filled and are only there to give no signal to market traders and analysts whether if the company is hiring or not.
The technical write up is great, but Mac users should not get too excited just yet on running 300B+ parameter models locally as the TPS isn't that good.
>...at 4.4+ tokens/second
That is even when it is using 4-bit quantization and it is still at that speed.
> The entire 209GB model streams from SSD through a custom Metal compute pipeline.
This is my main problem.
If I were to run this on a Mac SSD, 24/7 for heavy usage such as Openclaw, that is going to significantly reduce the lifetime of the SSD.
Can't imagine using this in the long term right now, but improvements will follow. Still a great write up anyways.
> If I were to run this on a Mac SSD, 24/7 for heavy usage such as Openclaw, that is going to significantly reduce the lifetime of the SSD.
How sure are you about that? I've never looked closer at how a large LLM with mixture of experts architecture switches between expert modules, but staying on roughly the same topic for the use (as it often would when editing the same codebase), I wouldn't be surprised to see the switches of composition are fairly rare, fairly small, and to the extent it happens it's repeated reads from the flash disk rather than writes it tends to cause.
Afaik the experts are not usually very interpretable, and generally would be surprised if at least one does not change every token. I don't know what happens in practice, but I know at least during training, nothing is done to minimize the number of expert switches between tokens.
I'd have thought at least a tiny explicit penalty term for switching, to discourage messing around with the composition without any expected gains from it.
If one is to use these on hardware that can't keep everything loaded I guess someone should examine how it works out in practice. Interpretability may be be a too much to ask, but I can't spontaneously see any reason why the experts can't at least be pushed to incorporate what's needed to remain the good choice for a longer segment.
The switching is done by layer, not just per token. Every layer is loading completely different parameters, you don't really benefit from continuity. You're generally better off shifting this work to the CPU, since CPU RAM is more abundant than the GPU's VRAM hence it matters less that so much of it is "wasted" on inactive expert layers. Disk storage is even more relatively abundant, so offloading experts to disk if you can't keep them in RAM (as OP does) is the next step.
Eh. I mean, 4 tokens a second works fine if you're patient. Go do something else while you wait.
I feel like whenever I'm trying to find information on which local models will work on my hardware, I have to overestimate because people don't know how to wait for things.
If you want decent throughput and do not care about burning SSD write cycles on a box that was never meant to act like a tiny inference server, a used server with actual RAM is still the cheaper and less silly option. I woudn't expect Apple's warranty team to be much help.
From "code" to "no-code" to "vibe coding" and back to "code".
What you are seeing here is that many are attempting to take shortcuts to building production-grade maintainable software with AI and now realizing that they have built their software on terrible architecture only to throw it away, rewriting it with now no-one truly understanding the code or can explain it.
We have a term for that already and it is called "comprehension debt". [0]
With the rise of over-reliance of agents, you will see "engineers" unable to explain technical decisions and will admit to having zero knowledge of what the agent has done.
This is exactly happening to engineers at AWS with Kiro causing outages [1] and now requiring engineers to manually review AI changes [2] (which slows them down even with AI).
> With the rise of over-reliance of agents, you will see "engineers" unable to explain technical decisions and will admit to having zero knowledge of what the agent has done.
I've had to work on multiple legacy systems like this where the original devs are long gone, there's no documentation, and everyone at the company admits it's complete mess. They send you off with a sympathetic, "Good luck, just do the best you can!"
I call it "throwing dye in the water." It's the opposite of fun programming.
On the other hand, it often takes creativity and general cleverness to get the app to do what you want with minimally-invasive code changes. So it should be the hardest for AI.
While I agree with everything you said, Amazon’s problems aren’t just Kiro messing up. It’s a brain drain due to layoffs, and then people quitting because of the continuous layoff culture.
While publicly they might say this is AI driven, I think that’s mostly BS.
Anyway, that doesn’t take away from your point, just adds additional context to the outages.
> We have a term for that already and it is called "comprehension debt".
This isn't any different than the "person who wrote it already doesn't work here any more".
> now requiring engineers to manually review AI changes [2] (which slows them down even with AI).
What does this say about the "code review" process if people cant understand the things they didn't write?
Maybe we have had the wrong hiring criteria. The "leet code", brain teaser (FAANG style) write some code interview might not have been the best filter for the sorts of people you need working in your org today.
Reading code, tooling up (debuggers, profilers), durable testing (Simulation, not unit) are the skill changes that NO ONE is talking about, and we have not been honing or hiring for.
No one is talking about requirements, problem scoping, how you rationalize and think about building things.
No one is talking about how your choice of dev environment is going to impact all of the above processes.
I see a lot of hype, and a lot of hate, but not a lot of the pragmatic middle.
I have been doing this a long time: my longest running piece of code was 20 years. My current is 10. Most of my code is long dead and replaced because businesses evolve, close, move on. A lot of my code was NEVER ment to be permanent. It solved a problem in a moment, it accomplished a task, fit for purpose and disposable (and riddled with cursing, manual loops and goofy exceptions just to get the job done).
Meanwhile I have seen a LOT of god awful code written by humans. Business running on things that are SO BAD that I still have shell shock that they ever worked.
AI is just a tool. It's going from hammers to nail guns. The people involved are still the ones who are ultimately accountable.
This sounded crazy in 2020 when I said that in [0]. Now it doesn't in 2026 and many have realized how unreliable GitHub has become.
If there was a prediction market on the next time GitHub would have at least one major outage per week, you would be making a lot of money since it appears that AI chatbots such as Tay.ai, Zoe and Copilot are somewhat in charge of wrecking the platform.
Any other platform wouldn't tolerate such outages.
[0] https://news.ycombinator.com/item?id=22867803
reply