FWIW - I also really like Wispr Flow, but I moved to running the 'Whisper Large' model locally using Handy (https://github.com/cjpais/Handy), which has been essentially as good, while also having lower latency.
Handy is great. It exposes a bunch of open models beyond Whipser too, and though I haven’t tried too many of them, I’ll throw in a rec for the Parakeet model which feels pretty much on par with Whisper for accuracy and is way way faster.
Up until a year ago I was regularly using a Massy Fergusson 135 [0] (Perkins Diesel version), made sometime in the 1970s. It was wonderful! So amazing to drive and use. Clunky and heavy, but you really really felt like you were using a machine. In low gears, if you put you foot down on the accelerator the engine would roar, and your speed would barely change!
And there was no fancy technology in it at all. If I was in the forest and had forgotten the key, I'd just reach behind the dashboard and hot-wire it. The air filter was basically a shisha-pipe that bubbled the incoming air through wire wool and engine oil.
Its fuel gauge didn't work either. You just had to take a look in the tank, or quickly react as soon as the revs started dropping. I ran it dry a few times and had to sit there with a spanner in one hand and YouTube into the other, while trying to bleed all the fuel lines. But they were all on the outside of the vehicle, which made it comparatively easy I imagine.
I've never actually driven a modern tractor, so don't know how it compares. I imagine the clutch is easier on the knees these days!
Anyway, this just felt like the place to share this.
> Up until a year ago I was regularly using a Massy Fergusson 135
There is a tradition in several European countries named Affouage: If you live in a rural area, you can get very cheap (or even free) wood at the condition that you go to cut it yourself in the close-by forest.
Many many people who are doing this practice are still using today Massy Fergusson 135, Renault R98/461, Ford 3000-4000 series, SOMECA or similar low tech tractors.
The reason are simple: They are cheap to operate, cheap to repair (damages happen easily forest environment) and their small size is perfect for the task.
The demand for these things will never die. Rugged environment requires cheap and robust hardware.
If this startup can capitalize on that, they do have a market.
I would like this in a new car company. Low tech, simple to operate, repair, etc. the tech is what goes out of style and I don’t know how (or don’t care) to use half of tech cars have to offer aside from the automatic safety stuff.
In most US states, if you're willing to build it yourself, you have a lot more leeway. It was part of the formula that allowed Local Motors to exist (sadly they aren't around now).
YMMV of course, but worth checking into if you have the mechanical and welding skills.
People put whatever engines they can find on old truck chassis. It’s up to you to do the swap. Most people do want to DIY anything on a car. Enthusiast based models usually don’t sell in high volume.
The hard part is convincing enough capitalists to invest into a high volume low margin product/company that doesn't have a real chance of being a jackpot win down the road. Investors don't want stocks that at its best gain a percentage or two above the overall market, there are plenty of other options with similar returns that have better worst-case results if there is a market crash. A co-op business could theoretically do it and would be happy with such margins, except it is unlikely a co-op is going to have enough capital to start with enough scale to actually push into the automotive market.
Sir Edmund Hillary and his team drove early Massey-Fergussons to the South Pole in 1958 (much to the annoyance of Sir Vivian Fuchs who Hillary beat to the pole even though Hillary was supposed to be a support player only :). The tractors weren't really suited to the job but they were tough enough to make it anyway.
>"But the stock of old repairable tractor will soon or later run dry."
Then some other country will come up with the alternative. It is insane that piece of very old and simple tech has such a price tag. There are whole bunch of countries that have cheap enough metal to come up with the alternative
The first motorized thing I've ever driven was a Massey Ferguson 37 I think. I still hope to have my own some day !
At the moment my little brother and my father, who are living in a rural area, have started a collection of vintage tractors by buying everything they can and fixing it up, help that my dad's a mechanic. I'm frankly jealous.
I learnt to drive on one of those. I'm a city kid but my grandfather was a wool farmer. Every school holiday we'd visit and I's spend my days literally puttering around the farm, which was pretty huge (~2000ha).
When I started out, 13ish or so, I had to stand on the clutch to get it down.
If you gave it enough beans and dropped the clutch it'll pop a wheelie! (Don't tell my grandpa)
Honestly, I still had to practically stand on the clutch with mine!
I'd teach someone to drive it and say, "now push down on the clutch". They they would heave and struggle, then eventually succeed and look victorious. I'd say, "well done, it is now half way down! But that's all you need for now!"
EDIT: To fully explain: It has a two-stage clutch. You half-press it and it disconnects the wheels from the engine. If you fully depress it all the way to the floor, it additionally disconnects the power-take-off shaft (PTO) from the engine. The PTO shaft is a spindle on the back of the tractor which drives things like your flail mower, wood chipper, etc.
EDIT 2: Edit 1 was for the general audience, not the parent commenter ;-)
Why was the clutch so heavy? Did it serve some purpose or was it just due to the limitations of the technology at the time?
I have certainly driven cars with lighter and heavier clutches (I live in EU, automatics weren't popular until recently and are still far from ubiquitous) but I couldn't tell you why every model just doesn't get a light clutch for comfort. A diesel Subaru I drove had a particularly heavy clutch as I recall, so at stop lights I would pop into neutral instead of holding the clutch down for an extended period.
To deliver very high torque, the clutch plates needs to be pressed very hard together to generate enough friction. This also means that it take a lot of force to pull them apart, if you use a simple lever, as older machines do.
Modern machines may use complex mechanical linkages to make the clutch easy to pull apart but still maintain a firm contact, but that also means higher cost and fragility. Or they use pneumatics or hydraulics to assist, sorta like power steering.
That, and design tolerances. A fancy clutch can be light and strong (think ferarri) but farm machines need to work in the dirt/rust and so need larger tolerances. So heavier springs and bigger .... Bigger everything. A slipping clutch in a Ferrari is annoying. A slipping clutch on a tractor means a missed harvest.
Plus mechanical release mechanisms of heavier machinery were often designed in a way that the clutch snaps at a certain point (also in order to reduce wear in the clutch).
Also because farm machines usually need max torque fast to break loose from static friction. You want a clutch to bite hard when pulling things through mud.
I once changed a broken release bearing of a truck. It was a relatively simple repair but the very heavy gearbox has to be taken out to do this - which is problematic especially if done on a yard without proper equipment.
Since then I always pop into neutral when standing at a traffic light. It is interesting how many people in manual driving cultures think there would be no wear and tear if they press the pedal down completely.
Of course there is, as there has to be a force translating connection between rotating parts and parts of the release mechanism which cannot rotate. Only when the pedal is left alone, the release bearing disconnects from the rotating clutch.
As a motorcyclist stopped at the traffic light I always keep the gear on and clutch pulled in. Why? Because I have to be ready to take off when the moron driver on the phone behind me fails to stop.
I mowed using a Farmall H on a family farm when I was about 12 y/o. I don't remember ever having deadly serious conversations with family members up to that point in my life. All four grandparents, aunts and uncles-- it seemed like everybody-- sat me down, looked me dead in the eye, and told me sternly and bluntly "you turn off the PTO and see the shaft isn't turning before you get off the tractor. Every. Time."
All of them knew somebody who lost an arm or leg or got killed when they got pulled into a PTO.
That was probably the first time I'd ever been given the opportunity to operate a machine that would fucking kill me if I shirked on respecting it. I will never forget the tone of that communication.
Without going too far into the weeds here, IMO this experience is representative of gun rights, zoning, and all sorts of other differences between urban and rural.
Rural kids are put into situations where they are expected to rely fully on themselves, with life-or-death consequences, from a young age. When your pre-teen is driving a machine on their own that could easily kill them or those around them, giving them a .22 rifle is just... normal. It's not at all the same situation as a kid the same age who lives in an apartment and who may have never been in a place where no one would be close enough to hear them if they screamed for help.
I can't wrap my head around the idea that a large number of people who live in cities seem to want to extend childhood through age 25. My daughters are 12 and 17, and between them have over fifty animals directly depending on them for survival. It's just... foreign.
I think you're generalizing too much. Rural communities take gun safety seriously. Farming communities take farming equipment seriously. Kids grow up internalizing the seriousness of these things, which is communicated expressly and tacitly their whole lives by countless people around them, including their friends. Plus they encounter walking examples of what can go wrong, like a missing finger, burn scars (not careful around bonfires or burn pits), or bullet holes (I knew at least 2 or 3 kids growing up with scars from shot). But put those same kids or adults who are careful with those machines in a similarly dangerous but novel situation, and they'll do dumb shit like anyone else. I'm tempted to argue they're more likely to do something dumb because they have a false confidence from their experience with other dangerous situations, whereas suburban and city kids may be more likely to be too scared to play around with any dangerous machine or situation.
I lived on a farm for a year as a young kid (farmer rented a couple of trailers on his land). I remember one day I was hanging around the hog pen watching the giant hogs mill about, probably contemplating trying to pet one. Mr Austin came by and sternly told me to not to reach through the fencing, then knelt down and showed me his ear, which was missing a big chunk.
On the flip side, plenty of Rural and Suburban people are terrified by the city, which kids growing up in the city shrug off.
Rural folks might learn to respect a PTO or the varmint rifle by age 10, but city kids learn how to navigate the bus routes and subway. They learn how to walk on crowded streets, how to live among a lot of different people, including dangerous people(and how to avoid the conflict).
It's all quite interesting. Different kinds of toughness, different kinds of mental fortitude.
I think that there's a major difference in the resulting mindsets that the two types of experiences form, though.
The first learn that nature is always present and doing its best to kill you / wreck your harvest, and that it is only through man's intelligence and social bonds that we thrive. I would argue a corollary of this is that one cannot tolerate malicious or grossly neglectful people around.
The second group learns that other people are a liability and that bad actors are just a fact of life to be tolerated and worked around.
Both approaches are clearly optimal for their respective environment. The former seems like a stronger foundation for building a civilization on, though.
This is becoming such a weird romanticisation of rural Americana!
Your civilisation is being destroyed because a largely rural constituency is able to clean a rifle in 60s but appears to have no critical thinking skills when it comes to a certain New Yorker.
Yes it’s good to learn how to be resilient in nature, but it’s also important to learn how to get along with and manage relationships with larger groups who are not always to be trusted.
The point missing from this discussion is that because of hysteria over stranger danger (not supported out by any real evaluation of or changes in risk) and because we allow cars to dominate our urban spaces, city kids are being denied opportunities for independence they previously had.
That’s the real change that’s happened … and we’re replacing real urban experience with corporate attention economies.
City kids can get on the bus or urban rail in actual big cities. Even in places like urban philippines or mexico where there is [often] no public transport, collectivos take up this niche. Kids abound in these places even in places like Manila where traffic is way worse and way more homicidal, and they take the jeepnee to go to the next barangay.
It's really mainly in the suburbs where neighborhoods are choked off by bike unfriendly freeways and no for-hire transit.
> The first learn that nature is always present and doing its best to kill you
> The second group learns that other people are a liability
Sounds like nature is simply survival + entropy and sometimes that leads to mixed incentives. Rural folks also understand people are dangerous. Per capita violent crime and murder is higher in Rural areas.
That's why I find it interesting, they're different expressions of common survival needs.
I grew up in a city, my wife on a ranch that was several sections large. We have lived in large dense cities and we currently live on a "smaller" ranch. I have taken densely packed subways to work on one hand and walked home on my own property more than a mile after running out of gas. Here is my observation. Neither camp typically has a clue about why the other might be motivated by a different opinion.
Take your .22 rifle. Many truly rural families would feel that this tool was essential. Same for having knife on you. We have lived where there are rattlesnake and coyotes as an almost every day thing. Not so much rattlesnakes but certainly coyotes. In fact bear and cougar were not out of the question. The idea that I would allow my kids to wander on the property without a .22 in their 4 wheeler seemed risky. They were expected to know how to shoot just as they were expected to know how to ride a horse, and drive a tractor unsupervised. We taught them to be safe and could not have run our ranch without our girls taking on some big dangerous responsibilities.
We have also lived in big cities where the idea that many of the liberties we enjoyed in the country were insane in the city. The idea that any random teen should be allowed to drive a 80hp tractor around or carry a gun or a fixed blade knife was insanity. Just as allowing my kids to run down the sidewalk or play unsupervised in the park after dark was insanity. On our fist day after moving my eldest daughter ran down the sidewalk and was hit (but not injured) by a car coming out of a driveway.
She just had no clue about how cars in a dense city moved. There just are different life rules that apply in different situations. Guns can be critically important in one environment and absolutely insane in a second. Same goes for driving a tractor that could kill you or a family member or going to a park after dark.
Unless people understand that a different environment might require a different set of norms or even laws we can't have a productive urban/rural conversation. Of course I can drive my ATV along your fence line. You probably can't even see it from your home or hear it. Though you can bet my dad asked your dad for permission 40 years ago. Try running your unregistered, unlicensed ATV through your suburban neighbor's yard and you will find out why there are important laws preventing you from doing what was perfectly fine in the back country.
I don't "want" to extend childhood; but where I live makes it a little difficult to let my kids roam the way I did. Go too far one way and you're heading into busy highway traffic hell, go too far the other way and you're heading into hobo territory.
Wish I could move; I could sell this overpriced place and almost retire.... not under my control
Family, yes. Nothing bad or negative, I like where I live a lot. It's just busy, and not super safe for wandering children, at least until they're a bit older.
People can have different lived experiences and it's OK; they are differently valuable and beneficial. I'm a certified unc, easily double the age of your oldest, and I have 0 animals depending on me for survival. It means, among other things, that I can simply decide to leave town for a week and don't need to arrange for replacement humans to take care of other living beings -- and this is a valuable freedom to have.
The phrasing of "gun rights" in the context that's really about gun responsibilities is a big part of the problem. And I say this from an unusual position; I'm a Brit who was taught to shoot at school (cadets). The urban gun control question is not so much about responsibility as about malice. There's not a huge number of people with murderous intent, but there are enough. And the resistance of rural America to the questions of either "do you actually need a gun?", "are you a responsible person?", and "no, you can't bring that into the city" result in thousands of deaths every year in the city. If they were willing to allow separate rules for different areas, this wouldn't be nearly as heated.
> a large number of people who live in cities seem to want to extend childhood through age 25
This is not great, and a more complicated problem of percieved danger.
Ive never heard a rural person talk about wanting to bring guns into cities, they generally avoid cities in general. I do hear lots of talk about urban focused gun regulations wanting to be passed nation wide without any exemptions for rural folks or even the slightest nuance or knowledge about guns and existing gun laws.
America's biggest obstacle to meaningful gun control is the groups and people pushing gun control the hardest have no idea what they are legislating. That's how we got nonsense bans on stuff like slings and bayonets, or trying to specifically ban AR-15s while functionally identical guns would remain legal.
And the second obstacle, which isn't far behind, is nobody trusts the US government enough to want to disarm themselves. We already got a police state and the largest military in the world and absolutely zero reasons in living memory to trust the government to look after its people instead of abusing them.
And frankly, I think any gun control measure in America is completely worthless and will only do more harm than good if the majority of gun owners don't trust the government enough to protect them all the time and aren't willingly turning in their guns. US citizens have enough guns to arm the entirety of themselves 100x times over so even if you could seize 95% of guns with magic, that's still enough blackmarket guns for 100+ years of insurgency with zero additional production. Personally I think we would do better to focus more on improving the lives of average citizens so they don't want to blast people randomly.
Maybe 'city' folk should offer something in exchange then. I would trade megamillion 'city' sovereignty on 'gun control' in exchange for stopping application of the NFA and GCA in the 'country'.
Right now all 'city' offers is a shittier deal and pray they do not alter it worse. Obviously that's not politically viable way to get agreement and part of the reason why gun control advocates think "nothing changes."
That's not what was done. The GCA and NFA was imposed on the entire country and none of the gun control advocates are offering to give that up in exchange for more local control. That was the "compromise." (oh yes, we did get an act allowing exemption from prosecution for merely travelling though a restrictive jurisdiction with an illegal weapon, but surprise surprise, places like New York just ignore that anyway and jail you until you can appeal it to federal court) It was always more and more regulation on people in the 'country' with nothing in exchange to offer them for having to give something up. And then, on top of that, the 'cities' added more on top of that (but refer to next paragraph for more).
When the 'country' finally got sick of it then you wound up with state pre-emption against local control being passed in most states because it turned out that bargain was a fraud.
So what I would propose, is if 'city' really wants to loosen up the gridlock, they should bring something serious to the negotiating table. Like ending the GCA and NFA in 'country' and in exchange state pre-emption gets nixed so 'city' can pass tighter laws there.
So to answer your question:
> Cant you all just pass some laws that apply to your place only? Why does it have to be a trade.
Here is where we are at. State pre-emption stops 'city' from passing stronger local control. And federal law stops 'country' from passing weaker local control. To break that gridlock 'city' and 'country' have to have something on the offering table for each other. That is why it has to be a trade.
>Rural kids are put into situations where they are expected to rely fully on themselves, with life-or-death consequences, from a young age.
come to the city, farm boy, and we'll give you a corner you can sling the brown from and we see how you do. we find something fo yo daughters to do too*
*i have absolutely no street smarts, country or city, but I do watch Law & Order and know how to pound a nail and know what to grease the maitre d' to get into the hottest restaurants in town. and beyond that i got friends, some of these guys know people who know people, just sayin
Perhaps one difference is that it seems to be considered good/normal parenting to expose your farm kids to mortal dangers, but it's definitely not considered good parenting to expose your city kids to mortal dangers (despite the city having no lack of mortal dangers).
I have never driven a tractor, but clearly remember our headmaster giving us this exact lecture when I was about 8. This in a town of 20,000 people where I expect not even 2% of the kids would even visit a farm outside of an organised trip, but clearly an important enough message to be worth broadcasting.
First farm death I ever knew of was related to me by a friend who used to live on the farm next to the dead person. He had PTO engaged but engine running while he was fixing a (I think) thresher. Clutch must have engaged and his dead, flailing remains were found by his wife when she came to call him in for dinner.
That seems to be common, the communist-era tractor I was riding was pretty much "stand with full weight and still have to brace by the steering wheel to push it"
Good that at least there wasn't much gear changing, pick one for task and just use it
My grandfather had one of these, though gas powered. It may have been the Ford model, cannot remember, though his was built I believe in the late 40s / early 50s. One story that still makes me laugh, he couldn't start it one day, and asked my grandmother to give him a pulling start w/ their ford diesel pickup. One look and my about 12 year old self just knew she wanted to be anywhere else but there (some foreshadowing, she had a reputation for a lead foot). Grandpa had already tied a rope from the tractor to the truck, and I believe he was in maybe one of the lower gears ready to pop the clutch after he got up to speed. Grandma tore (yes, tore) out of the yard shifting gears, and she was accelerating down their long driveway headed for the main road as Grandpa started frantically waving his hat trying to get her to stop. I'm pretty sure he never asked her again to help start the tractor. And yeah, the tractor was started, probably in the first 50 feet of that episode.
Did your dad grow up with subsistence farmers?
I also encountered plenty of hobby farmers who joke about the cost of their hobby which used to get me confused a bit since it was very much a food on the table thing for my grandparents and at least a small financial bonus to me.
Then again I don't buy much new other than seed. Everything feels like a ripoff nowadays.
Hey, we have the exact same model and it's still in use. Those things were built like tanks.
We have an even older one from the 1940s that also still fires up and works just fine.
The main thing the modern tractor buys at this point is simply a cabin with AC (which, TBH, is a VERY nice feature). Otherwise there's not much reason not to use the older farm equipment even today.
We'll still use the old massy for tilling and seeding.
I grew up on a farm in Germany. Our "little" tractor was the MF135 (3 cylinder engine!). I started driving it for real work at 12 for the simple reason that my grandpa, who was the third driver when the hay was being brought in, had passed away, and someone had to do it.
My dad and a cousin drove the big tractors. Can't remember whether MF55 or MF65, possibly one of each. Thundering monsters being driven flat out, double-clutched gear changes and all. The reason for all the rush is that the weather isn't that reliable in south Germany, and when they hay is dry and ready, it's all hands on deck.
Anyway... years later I visited the old homestead and there they still were, those big... umm. On Youtube you can probably still find a video of one being turned into a riding lawnmower, underslung mowing deck and all. Those tractors belong in the "Old MacDonald had a farm" era. The modern world is different with the base model of those tractors in the article having three times the power of those old ones, and it goes up from there.
We used to have a really old Massey Ferguson, I think TE-20, at the family (moonshine) farm. It was finally retired around 15 years ago and replaced with a MF 165. I hear you about the clutch--sometimes I feel I can't even push it down far enough.
I also love driving it, apart from the fact the hydraulics are somewhat off, so the front/rear lift won't ever stay in position.
The smaller tractors now mostly use a hydrostatic transmission instead of a clutch[]. You just move a plate that changes the mechanical advantage of the engine powered hydraulic drive. It's basically another set of hydraulics but for driving the tractor.
Still rocking one over here. The thing had not been maintained for 20 years while still being used, ran several times with almost no oil in the engine, drank gasoil full of water.
And it still works.
Things were made different back then.
I looked up the manual, you got everything you need to repair it. Maintenance is extremely easy. Even have electric schema.
Now my BMW, I looked into the manual how to change a light. It said to go to the dealer lol.
Fuck the modern car / tractor / tools. I blame the people for that, we went from customer that demanded to be able to repair their stuff to people who are now mechanically illiterate. I'm not sure they would even know how to replace a tire on their Tesla :)
That's why manufacturer have all the latitude to do what they do. And that's why it didn't go very far with farmers.
It's amazing we let it slip this far. Even cars from a decade or so ago feel much more repairable. I bought an EV and I haven't even seen the motor yet, because I'm going to have to dismantle a bunch of plastic-clipped stuff to remove the frunk, and I've broken enough brittle tabs for one lifetime. God forbid they'd just use actual metal fasteners for this stuff.
Go back to 1910, and more than 50% of the population lived in rural areas. And rural doesn't mean "suburbs". As this trend continues further back in time, I'd expect that people in their 30s may be living in cities in 1910, but we often not born there. They migrated from rural areas to the city.
Which means that city people even into the 50s had a very, very rural background.
So people who grew up on farms miles from any town or neighbours or stores, who had to rely upon themselves entirely, were the ones buying machines. But if you look at today, many people are apartment dwellers, or live in townhomes. They don't even have a place to fix something, let alone the tools or background.
I could fix any small engine before I was 10, work on cars before I could drive, and it's because you just picked up this stuff in a rural area. I guess my point is, if you don't know how to fix anything, and no one around you does except for specialists?
Then you probably won't care about owner repairability as much.
Sad, but probably a likely reason why we're where we are.
It's even worst tho, one day I layed a little bit against the front of the car and it made a reverse bump in the bodywork right on my ass.
Got a 2000 Suzuki that is full metal.
I think the trend of plastic went around 2000 to 2010 because of regulation on crash, plastic absorbs better the kinetic energy so we don't get our head smashed.
But yeah, no excuse to not make it easy to dismantle. It's the equivalent of Volkswagen using all kind of different screws to hold the plastic protection under the car, so that the average Joe who has standards screw drivers can't bleed his oil himself or change the gasoline filter.
This is maddening but you don't know it when you buy the car. It's only later.
Wild. We ran a 175 and 1100 for our daily tractors before Grandpa died and I quit farming (big ass John Deere machines for the real work at planting and harvest though).
They're phenomenal little machines that can do 99% of what you need. It blows my mind that for years, Grandpa farmed with a little Ford smaller than the 175. I can't imagine planting with that thing. The ww2 generation really were tough as nails.
The one I drove (and a much older MF as well) had both. A lever on the steering column, as well as a foot pedal. I've never seen anyone without one elsewhere either, maybe they were only sold that way in my country.
So our main small tractors were a 175 and an 1100. The 1100 had a bucket but I would've killed for a bucket on that little 175. Man that thing was handy. You could drive it through the yard without leaving tire tracks.
I shamefully have some Facebook Marketplace notifications for some Massy tractors. I'd love one. I don't even have land to use them, I just think they are neat.
I wonder if it's legal to just park your tractor in a regular parking spot across your apartment. I'm European so we have small parking spots. But would a small tractor fit in the parking spot of the biggest Ford truck?
My father drove one of those in his childhood. Now retired, he has bought a used one and uses it to maintain about an acre of land (and his grandkids love helping him).
Once, it broke down, and I was astonished to see that there are forums dedicated to this tractor. If I remember correctly, it was a problem with the fuel line that is rather common, and we managed to fix it thanks to these communities.
As I was researching it, I read stories of MF135s found abandoned in a ditch and starting immediately again. A robustness that makes this and other models popular in Africa...
I'm not sure the majority of the population will ever need, or even want, to learn to bleed fuel lines, so I wouldn't consider it reluctance. And I would wager that the majority of the (internet) population does engage in learning activities on the regular.
My son recently broke the string on the light cord in the bathroom. I opened it up in perhaps the naive expectation that someone would have designed that in such a way that the string can be reattached. Sadly it wasn't.
In fact when you open the interior plastic piece the whole thing springs apart and everything from the clicking mechanism to the electrical terminals explode in different directions.
Thankfully, someone had uploaded a video of a very similar switch and, after a few cross words (man I hate assembling mechanisms with springs), I had a new overhand knot in the string and all of the contacts, springs and terminals back in place.
I would, without doubt, drive down to a shop and buy a new one next time...
The ones here in the UK have these little plastic connectors on the string. The switch itself has a very short string coming out of it(<10cm), the plastic connector and then the main pull cord. These connectors are simple tubes with an opening that hold and hide the knots. Makes changing the pull cord quite easy, you just feed it through the hole in the connector, tie a simple knot at the end and pull it back into the connector body.
I actually had one of these connectors break on a bathroom light and just 3D printed a new one. But it should be fairly trivial to add one of these to any light pull you already have.
I think this kind of thing is much more commonplace than you think.
Never underestimate a young person and their phone. They not only use youtube or chatgpt to solve daily problems, but date, pay bills, and communicate with their friends using mostly videos/photos/emojis (and occasionally english).
New Zealand tree farmer Marty T has been posting detailed "back from the dead" tractor / bulldozer / grader / etc. restoration project videos for some time.
The name of the brand is "Massey Ferguson" not "Massy Fergusson".
The reason I know that is not that I'm a farmer. It's that 20 years ago a bunch of friends and I wrote and performed a parody song of Gainsbourg/Bardot song "Harley Davidson" where the motorbike brand was replaced with the tractor one.
> I imagine the clutch is easier on the knees these days!
Modern tractors don't really have a clutch. I mean they sorta do, but it's electronic. Even on sizable consumer positioned tractors(I have a JD 5055, but it applies to almost all the JD models), there's just a lever for forward, N, and reverse. Gear shifters work MUCH MUCH better now.
When I was younger I absolutely HATED changing gear on the tractor - it was a matter of dropping the revs which caused a dive, then a clunk finding the gear, then a jolt as the gear took hold and the revs came back up
Changing gears while driving? Are you sure you where supposed to? Many old tractors are without synced drives, so you are supposed to select gear before you start driving. Of course you can change when driving, but then you have to match revs to not get the drop betwen
I have the original 1940's Minneapolis Moline R and my wife has the original Farmall H and we both currently live in the city (but grew up farming or close to it) so we're not city kids, but somewhere stuck in between. I deeply get the feeling of using a non-tech machine, and how simple it is but intuitive to use. We used a pain mixing stick to check the gas level in our tractors on the farm, I don't think the gas gauges ever worked. You'd have to whack the starter with a wrench since they didn't ever work half the time. They worked over 60 years before they got their first oil change (my grandpa didn't believe in changing them - but my dad and I think it's just because you'll never get the canister filter to seal ever again if you did change it)
Those are so cool. First motorized thing I ever drove was some 1950s Ford tractor, as a little kid. My uncle showed me how to use it. I almost had to stand with both feet on the clutch and pull myself up to release it, while my brother manned the wheel and throttle separately.
I wasn't driving too much, but grew up with my uncle and grandfather using Ursus C-335, which is probably from the 70s. They said they traded an ox for it 40-50 years ago, still in use almost daily. Effectively zero electronics.
I went through my teenager years driving one of those MF 135 machines. A very versatile tractor. I enjoyed driving tractors (including a much older MF), when I eventually got my car's driver license some years later I found that driving cars weren't really that interesting.
During certain kinds of driving gear shifts became.. tricky. That's when I learned how to double-clutch, something I kept doing on cars as well, for many years after (think going steep uphill on snow and then having to shift into first gear without stopping)
My father still has one of these in orange and white. I remember when I was a little child and he would start it up, I could feel the concussion of the exhaust in my chest.
One of my early memories was driving a tractor like this hauling potato harvest with my late grandfather when his "big" tractor wouldn't start. Feels like a 1000 years ago...
Thanks for sharing all those experiences and tech explanations in how tractor clutches work everyone.
Great thread to read. I'm kinda surprised the OP made the HN tops but DIY anything seems to be experiencing a Renaissance. (Not too big a surprise in HN I suppose)
You say “up until a year ago”, what ended up replacing it?
I’m in the market for a tractor in roughly that size, and am very tempted to just find an old machine in decent shape. I’d be very curious about the decision/experience if you did upgrade to something more modern?
The old machines are rock solid. They really just keep going and going.
Really the only reason we got a new tractor was for the cabin with AC. That does make it a lot easier to spend an entire day working field. Otherwise you have to deal with whatever weather it is outdoors which can be unpleasant if it's very cold or very hot.
I loved the MF 135 my neighbour had. It was great. The injector pump had failed and we'd swapped it with one off a marine version of the Perkins AD3, which had a reasonably "opened up" governor on it. Flat it out could do a whopping 20mph!
Any technology from before the time of your grandparents, and often parents, is usually perceived to be "not fancy". Because then those elders can't tell you in your childhood what life was like before that technology. So in your lived experience that technology was always there. Reading history later on, doesn't change your emotional experiences.
Disagree. There's lots of products and goods that have become less fancy as a result of changes in labor/material cost as industrialization ran its course and the old way is considered the fancy way.
Wood furniture joined with glue and pegs rather than inserts and screws. Solid wood furniture at all. Leather and natural fibers gave way to plastics. Ornate castings gave way to simple stampings and simply castings (where things are still cast).
An internal combustion engine may be complex, but it's not fancy. I can see and touch and understand every part of it. I can maintain and modify and repair it. This is not true for fancy electronics and certainly not locked-down proprietary firmware.
For every modern car I had or used in the last 20 years, the engine itself never was a problem, other than the regular maint, oil, filters, belts, plugs, cables...
Now, electronics problems, albeit relativelly rare, were far more common and fucking expensive.
And then, but this more due to the state of modern roads and streets than the car themselves, suspension issues.
The magic of an engine is less in how it operates, and more in how it was built. At least around the time they started showing up, manufacturing lots of precision metal parts was not trivial.
Although modern electronics take this further, with both operation and construction being utterly complex.
One of my vehicles is a 2009 Civic. It continues to amaze me that with minimal maintenance, that 17-year-old vehicle will fire right up with the turn of a key, with hundreds (thousands?) of parts moving in a specific way, many of them with tolerances in tiny fractions of an inch.
2010 MB C300 I bought in 2013 from a dealer after the lease expired, parked outside without a garage or cover since then (Virginia).
About 3 years ago a large branch (about 8" diameter) from an old overhanging tree fell right on the transparent sunroof cover and shattered it into a million pieces. After picking them out of the sunroof mechanism (which no longer worked after the impact) and the inside of the car, I covered the opening with several sheets of magnetized vinyl. Works great, never a drop of water inside since then and it's stayed in place without any attention. Temperature control inside the car at rest or while driving at highway speed is like it was before the damage.
Being old now I never go anywhere since I can get stuff delivered. About every 3 weeks I go out and the car starts right up, I drive a 5-mile loop to circulate the oil and then park it for another 3 weeks. Been doing this for years. I do get an oil change annually.
I learned how engines worked by taking apart, cleaning and reassembling an ancient lawnmower engine so I could use it on my go-kart. I then learned how cars worked by taking one apart and putting it back together again.
Neither of those machines had a transistor in them. It was all basic electricity.
Wait a few years and no HD will be able to do something similar.
See other story on front page right now: educational scores are trending down and that trend is only going to accelerate now that every student is using LLMs.
This is something we've[0] done a number of times for customers coming from various cloud providers. In our case we move customers onto a multi-server (sometimes multi-AZ) deployment in Hetzner, using Kubernetes to distribute workloads across servers and provide HA. Kubernetes is likely a lot for a single node deployment such as the OP, but it makes a lot more sense as soon as multiple nodes are involved.
For backups we use both Velero and application-level backup for critical workloads (i.e. Postgres WAL backups for PITR). We also ensure all state is on at least two nodes for HA.
We also find bare metal to be a lot more performant in general. Compared to AWS we typically see service response times halve. It is not that virtualisation inherently has that much overhead, rather it is everything else. Eg, bare metal offers:
- Reduced disk latency (NVMe vs network block storage)
- Reduced network latency (we run dedicated fibre, so inter-az is about 1/10th the latency)
- Less cache contention, etc [1]
Anyway, if you want to chat about this sometime just ping me an email: adam@ company domain.
> We also find bare metal to be a lot more performant in general
I measured this several years back and never looked at virtual servers again. Since CPU time isn't reserved (like RAM is), the performance is abysmal compared to real hardware.
I haven't benchmarked recently, but I believe the situation is roughly still the same. If anything, things might have gotten worse because of more overbooking going on.
Moving around k8s deployments is really nice. Very little vendor lockin compared to many of the cloud things you can buy.
My entire stack is.. k8s, hosted Postgres, s3 type storage. I can always host my own Postgres. So really down to k8s and s3. I think hetzner has some kind of s3 storage but haven’t looked into, and I assume moving in 100 TB is a process….
We can run a Forgejo instance for you with Firecracker VM runners on bare metal. We can also support it and provide an SLA. We're running it internally and it is very solid. We're running the runners on bare metal, with a whole lot of large CI/CD jobs (mostly Rust compilation).
The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!
I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.
I actually went through some of the issue/pr stuff for the forgejo project after I asked you. It seems like things are moving along nicely and you seem to have found a welcoming environment in their repo. I will keep an eye on that progress. Thanks very much. I do not have a pressing need but firecracker runners would be pretty awesome to have.
We've migrated to Forgejo over the last couple of weeks. We position ourselves[0] as an alternative to the big cloud providers, so it seemed very silly that a critical piece of our own infrastructure could be taken out by a GitHub or Azure outage.
It has been a pretty smooth process. Although we have done a couple of pieces of custom development:
1) We've created a Firecracker-based runner, which will run CI jobs in Firecracker VMs. This brings the Foregjo Actions running experience much more closely into line with GitHub's environment (VM, rather than container). We hope to contribute this back shortly, but also drop me a message if this is of interest.
2) We're working up a proposal[1] to add environments and variable groups to Forgejo Actions. This is something we expect to need for some upcoming compliance requirements.
I really like Forgejo as a project, and I've found the community to be very welcoming. I'm really hoping to see it grow and flourish :D
This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between:
1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based).
2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend.
3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time).
4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.
A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system.
Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team!
I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run.
A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours.
This 'microservices' push usually means that instead of having an on-server session where you can serve stuff from a temporary cache, all the data that persists between requests needs to be stored in a db somewhere, all the auth logic needs to re-check your credentials, and something needs to direct the traffic and load balance these endpoint, and all this stuff costs money.
I think if you have 4 Java boxes as servers with a redundant DB with read replicas on EC2, your infra is so efficient and cheap that even paying 4x for it rather than going for colocation is well worth it because of the QoL and QoS.
These crazy AWS bills usually come from using every service under the sun.
The complexity is what gets you. One of AWS's favorite situations is
1) Senior engineer starts on AWS
2) Senior engineer leaves because our industry does not value longevity or loyalty at all whatsoever (not saying it should, just observing that it doesn't)
3) New engineer comes in and panics
4) Ends up using a "managed service" to relieve the panic
5) New engineer leaves
6) Second new engineer comes in and not only panics but
outright needs help
7) Paired with some "certified AWS partner" who claims to help "reduce cost" but who actually gets a kickback from the extra spend they induce (usually 10% if I'm not mistaken)
Calling it it ransomware is obviously hyperbolic but there are definitely some parallels one could draw
On top of it all, AWS pricing is about to massively go up due to the RAM price increase. There's no way it can't since AWS is over half of Amazon's profit while only around 15% of its revenue.
One of the biggest problems with the self-hosted situations I’ve seen is when the senior engineers who set it up leave and the next generation has to figure out how to run it all.
In theory with perfect documentation they’d have a good head start to learn it, but there is always a lot of unwritten knowledge involved in managing an inherited setup.
With AWS the knowledge is at least transferable and you can find people who have worked with that exact thing before.
Engineers also leave for a lot of reasons. Even highly paid engineers go off and retire, change to a job for more novelty, or decide to try starting their own business.
unfortunately it lot of things in AWS that also could be messed up so it might be really hard to research what is going on. For example, you could have hundreds of Lambdas running without any idea where original sources and how they connected to each-other, or complex VPCs network routing where some rules and security groups shared randomly between services so if you do small change it could lead to completely difference service to degrade (like you were hired to help with service X but after you changes some service Y went down and you even not aware that it existed)
Not much different from how it worked in companies I used to work for. Except the situation was even worse as we had no api or UI to probe for information.
The end result of all this is that the percentage of people who know how to implement systems without AWS/Azure will be a single digit. From that point on, this will be the only "economic" way, it doesn't matter what the prices are.
That's not a factual statement over reality, but more of a normative judgement to justify resignation. Yes, professionals that know how to actually do these things are not abundantly available, but available enough to achieve the transition. The talent exists and is absolutely passionate about software freedom and hence highly intrinsically motivated to work on it. The only thing that is lacking so far is the demand and the talent available will skyrocket, when the market starts demanding it.
They actually are abundantly available and many are looking for work. The volume of "enterprise IT" sysadmin labor dwarfs that of the population of "big tech" employees and cloud architects.
I've worked with many "enterprise IT" sysadmins (in healthcare, specifically). Some are very proficient generalists, but most (in my experience) are fluent in only their specific platforms, no different than the typical AWS engineer.
Perhaps we need bootcamps for on prem stacks if we are concerned about a skills gap. This is no different imho from the trades skills shortage many developed countries face. The muscle must be flexed. Otherwise, you will be held captive by a provider "who does it all for you".
"Today, we are going to calculate the power requirements for this rack, rack the equipment, wire power and network up, and learn how to use PXE and iLO to get from zero to operational."
This might be my own ego talking (I see myself as a generalist), but IMHO what we need are people that are comfortable jumping into unfamiliar systems and learning on-the-fly, applying their existing knowledge to new domains (while recognizing the assumptions their existing knowledge is causing them to make). That seems much harder to teach, especially in a boot camp format.
As a very curious autodidact, I strongly agree, but this talent is rare and can punch it's own ticket (broadly speaking). These people innovate and build systems for others to maintain, in my experience. But, to your point, we should figure out the sorting hat for folks who want to radically own these on prem systems [1] if they are needed.
I don't really think so. That was a ship that sailed ten years ago and nearly every sysadmin who is still proficient with managing on-prem stacks has adapted to also learn how to manage VPCs in an arbitrary cloud. It's not like this is a recent change.
> and the talent available will skyrocket, when the market starts demanding it.
Part of what clouds are selling is experience. A "cloud admin" bootcamp graduate can be a useful "cloud engineer", but it takes some serious years of experience to become a talented on prem sre. So it becomes an ouroboros: moving towards clouds makes it easier to move to the clouds.
> A "cloud admin" bootcamp graduate can be a useful "cloud engineer",
If by useful you mean "useful at generating revenue for AWS or GCP" then sure, I agree.
These certificates and bootcamps are roughly equivalent to the Cisco CCNA certificate and training courses back in the 90's. That certificate existed to sell more Cisco gear - and Cisco outright admitted this at the time.
In part - yes. Useful as in capable of spinning up services without opening glaring security holes or bringing half of the infra down. Like with any tech, it takes experience and guardrails to use it efficiently and effectively.
> A "cloud admin" bootcamp graduate can be a useful "cloud engineer"
That is not true. It takes a lot more than a bootcamp to be useful in this space, unless your definition is to copy-paste some CDK without knowing what it does.
Only time will tell. It depends on when someone with a MBA starts asking questions about cloud spending and runs the real numbers. People promoting self hosting often are not counting all the cost of self hosting (AWS has people working 24x7 so that if something fails someone is there to take action)
> AWS has people working 24x7 so that if something fails someone is there to take action..
The number of things that these 24x7 people from AWS will cover for you is small. If your application craps out for any number of reasons that doesn't have anything to do with AWS, that is on you. If your app needs to run 24x7 and it is critical, then you need your own 24x7 person anyway.
All the hardware and network issues are on them. I agree that you still need your own people to support you applications, but that is only part of the problem.
I've got thousands of devices over hundreds of sites in dozens of countries. The number of hardware failures are a tiny number, and certainly don't need 24/7 response
From what I've seen, if you're depending on AWS, if something fails you too need someone 24x7 so that you can take action as well. Sometimes magic happens and systems recover after aws restarts their DNS, but usually the combination of event causes the application to get into an unrecoverable state that you need manual action. It doesn't always happen but you need someone to be there if it ever happens. Or bare minimum you need to evaluate if the underlying issue is really caused by AWS or something else has to be done on top of waiting for them to fix.
I've only had one outage I could attribute to running on-prem, meanwhile it's a bit of a joke with the non-IT staff in the office that when "The Internet" (i.e. Cloudflare, Amazon) goes down with news reports etc our own services are all running fine.
Distributed systems can partly fail in many subtly different ways, and you almost never notice it because there are people on-call taking care of them.
It already is like that, but not because of the cloud. Those of us who begun with computers in the era of the command line were forced to learn the internals of operating systems, and many ended up turning this hobby into a job.
Youngsters nowadays start with very polished interfaces and smartphones, so even if the cloud wasn't there it would take them a decade to learn systems design on-the-job, which means it wouldn't happen anyway for most. The cloud nowadays mostly exists because of that dearth of system internals knowledge.
While there still are around people who are able to design from scratch and operate outside a cloud, these people tend to be quite expensive and many (most?) tend to work for the cloud companies themselves or SaaS businesses, which means there's a great mismatch between demand and supply of experienced system engineers, at least for the salaries that lower tier companies are willing to pay. And this is only going to get worse. Every year, many more experienced engineers are retiring than the noobs starting on the path of systems engineering.
It’s all anecdotal but in my experiences it’s usually opposite. Bored senior engineer wants to use something new and picks a AWS bespoke service for a new project.
I am sure it happens a multitude of ways but I have never seen the case you are describing.
I’ve seen your case more than the ransom scenario too. But also even more often: early-to-mid-career dev saw a cloud pattern trending online, heard it was a new “best practice,” and so needed to find a way to move their company to using it.
Is that what I should be doing? I'm just encouraging the devs on my team to read designing data intensive apps and setting up time for group discussions. Aside from coding and meetings that is.
> 4) Ends up using a "managed service" to relieve the panic
It's not as though this is unique to cloud.
I've seen multiple managers come in and introduce some SaaS because it fills a gap in their own understanding and abilities. Then when they leave, everyone stops using it and the account is cancelled.
The difference with cloud is that it tends to be more central to the operation, so can't just be canceled when an advocate leaves.
I'll give you an alternative scenario, which IME is more realistic.
I'm a software developer, and I've worked at several companies, big and small and in-between, with poor to abysmal IT/operations. I've introduced and/or advocated cloud at all of them.
The idea that it's "more expensive" is nonsense in these situations. Calculate the cost of the IT/operations incompetence, and the cost of the slowness of getting anything done, and cloud is cheap.
Extremely cheap.
Not only that, it can increase shipping velocity, and enable all kinds of important capabilities that the business otherwise just wouldn't have, or would struggle to implement.
Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers.
> Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers
This has been my experience as well. There are legitimate points of criticism but every time I’ve seen someone try to make that argument it’s been comparing significantly different levels of service (e.g. a storage comparison equating S3 with tape) or leaving out entire categories of cost like the time someone tried to say their bare metal costs for a two server database cluster was comparable to RDS despite not even having things like power or backups.
That leaves out staffing, backups, development and testing of a multi-location failover mechanism as robust as the RDS one, and a bunch of security compliance work if that’s relevant.
It’s totally possible to beat AWS and volume is the way to do it–your admin’s salary doesn’t scale be linearly with storage–but every time I’ve tried to account for all of the costs it’s been close enough that it’s made sense to put people on things which can’t be outsourced.
If this database is a large portion of the infrastructure required then the fixed-ish costs don't scale so well, but a smaller cloud/hosting company should be considered.
But I have over 60 servers. Using the pricing calculator for the two AWS SaaS services that closely align with our primary service (40+ of those servers), we'd face a cost of over $1.2M/year if reserved for 3 years and paid upfront — that's for the service alone, I haven't added any bandwidth costs, or getting the data into those systems, and I've picked the minimum values for storage and throughput as I don't know what these should be. (Probably not the minimum.)
Add the remaining compute (~20 decent servers), a petabyte-scale storage pool, and all the rest, and the bill would likely exceed our entire IT budget including hardware, hosting, cloud services we do use, and all the salaries.
My rough estimate is our infrastructure costs would increase 8-10 times using AWS, our staff costs wouldn't reduce, and the risk to the budget would increase with variable usage.
This is tax money being spent, so I am asked every few years to justify why we aren't using cloud. (That's why I'm putting this much effort into a HN reply, the question was asked again recently.)
I know someone working in another country on essentially the same system for that country. They went all-in on AWS and pay every 1-2 months what we spend in a year, but have a fraction of our population/data.
Fron what I've seen this can work as a stopgap until IT get their hooks into the cloud system in which case you circle back to paying to costs of incompetence and the costs of the cloud (sometimes stacking on top of each other).
There's still a benefit in terms of infrastructure reliability. Recovery times are faster, backups more reliable, etc. Basically, vendor managed is better than customer managed in most situations, assuming a competent vendor.
Also, if the cloud systems are architected properly before IT gets hold of them, then they tend to retain their good properties for a long time, especially if others are paying attention to e.g. gitops pull requests.
My current company ended up replacing its (small) operations team in order to get people with cloud expertise. We hired the new team for the skills we needed. It's worked out well.
Just this week a friend of mine was spinning up some AWS managed service, complaining about the complexity, and how any reconfiguration took 45 minutes to reload. It's a service you can just install with apt, the default configuration is fine. Not only is many service no longer cheaper in the cloud, the management overhead also exceed that of on-prem.
I'd gladly use (and maybe even pay for!) an open-source reimplementation of AWS RDS Aurora. All the bells and whistles with failover, clustering, volume-based snaps, cross-region replication, metrics etc.
As far as I know, nothing comes close to Aurora functionality. Even in vibecoding world. No, 'apt-get install postgres' is not enough.
serverless v2 is one of the products that i was skeptical about but is genuinely one of the most robust solutions out there in that space. it has its warts, but I usually default to it for fresh installs because you get so much out of the box with it
Nitpick (I blame Amazon for their horrible naming): Aurora and RDS are separate products.
What you’re asking for can mostly be pieced together, but no, it doesn’t exist as-is.
Failover: this has been a thing for a long time. Set up a synchronous standby, then add a monitoring job that checks heartbeats and promotes the standby when needed. Optionally use something like heartbeat to have a floating IP that gets swapped on failover, or handle routing with pgbouncer / pgcat etc. instead. Alternatively, use pg_auto_failover, which does all of this for you.
Clustering: you mean read replicas?
Volume-based snaps: assuming you mean CoW snapshots, that’s a filesystem implementation detail. Use ZFS (or btrfs, but I wouldn’t, personally). Or Ceph if you need a distributed storage solution, but I would definitely not try to run Ceph in prod unless you really, really know what you’re doing. Lightbits is another solution, but it isn’t free (as in beer).
Cross-region replication: this is just replication? It doesn’t matter where the other node[s] are, as long as they’re reachable, and you’ve accepted the tradeoffs of latency (synchronous standbys) or potential data loss (async standbys).
Metrics: Percona Monitoring & Management if you want a dedicated DB-first, all-in-one monitoring solution, otherwise set up your own scrapers and dashboards in whatever you’d like.
What you will not get from this is Aurora’s shared cluster volume. I personally think that’s a good thing, because I think separating compute from storage is a terrible tradeoff for performance, but YMMV. What that means is you need to manage disk utilization and capacity, as well as properly designing your failure domain. For example, if you have a synchronous standby, you may decide that you don’t care if a disk dies, so no messing with any kind of RAID (though you’d then miss out on ZFS’ auto-repair from bad checksums). As long as this aligns with your failure domain model, it’s fine - you might have separate physical disks, but co-locate the Postgres instances in a single physical server (…don’t), or you might require separate servers, or separate racks, or separate data centers, etc.
tl;dr you can fairly closely replicate the experience of Aurora, but you’ll need to know what you’re doing. And frankly, if you don’t, even if someone built a OSS product that does all of this, you shouldn’t be running it in prod - how will you fix issues when they crop up?
> you can fairly closely replicate the experience of Aurora
Nobody doubts one could build something similar to Aurora given enough budget, time, and skills.
But that's not replicating the experience of Aurora. The experience of Aurora is I can have all of that, in like 30 lines of terraform and a few minutes. And then I don't need to worry about managing the zpools, I don't need to ensure the heartbeats are working fine, I don't need to worry about hardware failures (to a large extent), I don't need to drive to multiple different physical locations to set up the hardware, I don't need to worry about handling patching, etc.
You might replicate the features, but you're not replicating the experience.
The person I replied to said they wanted an open-source reimplementation of Aurora. My point - which was probably poorly-worded, or just implied - was that there's a lot of work that goes into something like that, and if you can't put the pieces together on your own, you probably shouldn't be running it for anything you can't afford downtime on.
Managed services have a clear value proposition. I personally think they're grossly overpriced, but I understand the appeal. Asking for that experience but also free / cheap doesn't make any sense.
> Asking for that experience but also free / cheap doesn't make any sense.
Things that used to be very expensive suddenly gets available for free after someone builds an open source version of it. That's just the nature of open source.
It's unreasonable to demand it from someone, but people do build things and release them for free all the time! Indeed, it makes plenty of sense to imagine that at some point in time, open source offerings of Postgres will be comparable to Aurora in ease of use.
What managed service? Curious, I don’t use the full suite of aws services but wondering what would take 45mins, maybe it was a large cluster of some sort that needed rolling changes?
My observation is that all these services are exploding in complexity, and they justify saying that there are more features now, so everyone needs to accept spending more and more time and effort for the same results.
It's basically the same dynamic as hedonic adjustment in the CPI calculations. Cars may cost twice as much now they have usb chargers built in so inflation isn't really that bad.
> If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours.
If ECS is faster, then you're more satisfied with AWS and less likely to migrate. You're also open to additional services that might bring up the spend (e.g. ECS Container Insights or X-Ray)
We did some benchmarks and ECS was definitely quite a bit more expensive for a given capacity than just running docker on our own EC2 instances. It also bears pointing out that a lot of applications (either in-house or off-the-shelf) expect a persistent mutable config directory or sqlite database.
We used EFS to solve that issue, but it was very awkward, expensive and slow, its certainly not meant for that.
Fully agree to this. I find the cost of cloud providers is mostly driven by architecture. If you're cost conscious, cloud architectures need to be up-front designed with this in mind.
Microservices is a killer with cost. For each microservices pod
- you're often running a bunch of side cars - datadog, auth, ingress
- you pay massive workload separation overhead with orchestration, management, monitoring and ofc complexity
I am just flabbergasted that this is how we operate as a norm in our industry.
I don’t understand why most cloud backend designs seem to strive for maximizing the number of services used.
My biggest gripe with this is async tasks where the app does numerous hijinks to avoid a 10 minute lambda processing timeout. Rather than structure the process to process many independent and small batches, or simply using a modest container to do the job in a single shot - a myriad of intermediate steps are introduced to write data to dynamo/s3/kinesis + sqs/and coordination.
A dynamically provisioned, serverless container with 24 cores and 64 GB of memory can happily process GBs of data transformations.
It's about fitting your utilization to the model that best serves you.
If you can keep 4 "Java boxes" fed with work 80%+ of the time, then sure EC2 is a good fit.
We do a lot of batch processing and save money over having EC2 boxes always on. Sure we could probably pinch some more pennies if we managed the EC2 box uptime and figured out mechanisms for load balancing the batches... But that's engineering time we just don't really care to spend when ECS nets us most of the savings advantage and is simple to reason about and use.
Also don’t forget the layers of REST services, ORMs, and other lunacy.
It runs slower than a bloated pig, especially on a shared hosting node, so now needs Kubernetes and cloud orchestration to make it “scalable” - beyond a few requests per second.
Agreed. There is a wide price difference between running a managed AWS or Azure MySQL service and running MySQL on a VM that you spin up in AWS or Azure.
Great comment. I agree it's a spectrum and those of us who are comfortable on (4) like yourself and probably us at Carolina Cloud [0] as well, (4) seems like a no brainer. But there's a long tail of semi-technical users who are more comfortable in 2-3 or even 1, which is what ultimately traps them into the ransomware-adjacent situation that is a lot of the modern public cloud. I would push back on "usage-based". Yes it is technically usage-based but the base fee also goes way up and there are also sometimes retainers on these services (ie minimum spend). So of course "usage-based" is not wrong but what it usually means is "more expensive and potentially far more expensive".
The problem is that clouds have easily become 3 or 5 times the price of managed services, 10x the price of option 3, and 20x the price of option 4. To say nothing of the fact that almost all businesses can run fine on "pc under desk" type situations.
So in practice cloud has become the more expensive option the second your spend goes over the price of 1 engineer.
Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.
> I’m a bit scared of managing the services on my own
I see it from the other direction, when if something fails, I have complete access to everything, meaning that I have a chance of fixing it. That's down to hardware even. Things get abstracted away, hidden behind APIs and data lives beyond my reach, when I run stuff in the cloud.
Security and regular mistakes are much the same in the cloud, but I then have to layer whatever complications the cloud provide comes with on top. If cost has to be much much lower if I'm going to trust a cloud provider over running something in my own data center.
You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours.
We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings"
(plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)
I've just shifted my SWE infrastructure from AWS to Hetzner (literally in the last month). My current analysis looks like it will be about 15-20% of the cost - £240 vs 40-50 euros.
Expect a significant exit expense, though, especially if you are shifting large volumes of S3 data. That's been our biggest expense. I've moved this to Wasabi at about 8 euros a month (vs about $70-80 a month on S3), but I've paid transit fees of about $180 - and it was more expensive because I used DataSync.
Retrospectively, I should have just DIYed the transfer, but maybe others can benefit from my error...
Extremely useful information - unfortunately I just assumed this didn't apply to me because I am in the UK and not the EU. Another mistake, though given it's not huge amounts of money I will chalk it up to experience.
Hopefully someone else will benefit from this helpful advice.
I’m curious to know the answer, too. I used to deploy my software on-prem back in the day, and that always included an installation of Microsoft SQL Server. So, all of my clients had at least one database server they had to keep operational. Most of those clients didn’t have an IT staff at all, so if something went wrong (which was exceedingly rare), they’d call me and I’d walk them through diagnosing and fixing things, or I’d Remote Desktop into the server if their firewalls permitted and fix it myself. Backups were automated and would produce an alert if they failed to verify.
It’s not rocket science, especially when you’re talking about small amounts of data (small credit union systems in my example).
No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.
Heroku was popular with startups who didn’t have infrastructure skills but the price was high enough that anyone who wasn’t in that triangle of “lavish budget, small team, limited app diversity” wasn’t using it. Things like AWS IaaS were far more popular due to the lower cost and greater flexibility but even that was far from a majority service class.
I am not sure if you are trying to refute my lived experience or what exactly the point is. Heroku was wildly popular with startups at the time, not just those with lavish budgets. I was already touching RDS at this point and even before RDS came around no organization I worked at had me jumping on bare metal to provision services myself. There always a system in place where someone helped out engineering to deploy systems. I know this was not always the case but the person I was responding to made it sound like 15 years ago all engineers were provisioning their own database and doing other times of dev/sys ops on a regular basis. It’s not true at least in SV.
I have no doubt that was your experience. My point was that it wasn’t even common in SV as whole, just the startup scene. Think about headcount: how many times fewer people worked at your startup than any one of Apple, Oracle, HP, Salesforce, Intuit, eBay, Yahoo, etc.? Then thing about how many other companies there are just in the Bay Area who have large IT investments even if they’re not tech companies.
Even at their peak, Heroku was a niche. If you’d gone conferences like WWDC or Pycon at the time, they’d be well represented, yes, and plenty of people liked them but it wasn’t a secret that they didn’t cover everyone’s needs or that pricing was off putting for many people, and that tended to go up the bigger the company you talked to because larger organizations have more complex needs and they use enough stuff that they already have teams of people with those skills.
I think we are talking past each other here. While your language is a bit proactive originally your not wrong as I have already agreed startups absolutely, lavish budgets? No that’s just silly.
Again 15 years even in moderately large organizations it was quite common as a product engineer to not be responsible for the provisioning all the required services for whatever you were building. And again it’s not the rule but it is far from being an exception. Not sure what you’re trying to prove or disprove.
A tricky thing on this site is that there are lots of different people with very different kinds of experience, which often results in people talking past each other. A lot of people here have experience as zero-to-one early startup engineers, and yep, I share your experience that Heroku was very popular in that space. A lot of other people have experience at later growth and infrastructure focused startups, and they have totally different experiences. And other people have experience as SREs at big tech, or doing IT / infrastructure for non-tech fortune 500 businesses. All of these are very different experiences, and very different things have been popular over the last couple decades depending on which kind of experience you have.
Absolutely true but I also think it’s a fair callout when the intent was to disprove the original post asking how old someone was because 15 years ago everyone was stringing together their own services which is absolutely not true. There were many shades of gray at that time both in my experience of either have a sysops/devops team to help or deploying to Heroku as well as folks that were indeed stringing together services.
I find it equally disingenuous to suggest that Heroku was only for startups with lavish budgets. Absolutely not true. That’s my only purpose here. Everyone has different experiences but don’t go and push your own narrative as the only one especially when it’s not true.
I kind of thought the "15 years" was just one of those things where people kind of forget what year it is. Wow, 2010 was already over 15 years ago?? That kind of mistake. I think this person was thinking pre-2005. I graduated college just after that, and that's when all this cloud and managed services stuff was just starting to explode. I think it's true that before that, pretty much everyone was maintaining actual servers somewhere. (For instance, I helped out with the physical servers for our CS lab some when I was in college. Most of what we hosted on those would be easier to do on the cloud now, but that wasn't a thing then.)
Did you fail to finish reading the rest? At the same time I had touch with organizations that were still in data centers but I as an engineer had no touch on the bare metal and ticket systems were in place to help provision necessary services. I was not deploying my own Postgres database.
It's 2026 and banks are still running their mainframe, running windows VMs on VMware and building their enterprise software with Java.
The big boys still have their own datacenters they own.
Sure, they try dabbling with cloud services, and maybe they've pushed their edge out there, and some minor services they can afford to experiment with.
If you are working at a bank you are most likely not standing up your own Postgres and related services. Even 15 years ago. I am not saying it never happened, I am saying that even 15 years ago even large orgs with data enters often had in place sys and devops that helped with providing resources. Obviously not the rule but also not an exception.
True. We had separate teams for Oracle and MSSQL management. We had 3 teams each for Windows, "midrange" (Unix) and mainframe server management. That doesn't include IAM.
Ahah I'm 31, but deciding if it makes sense to manage your own db doesn't depend on the age of the CTO.
See, turning up a VM, installing and running Postgres is easy.
The hard part is keeping it updated, keeping the OS updated, automate backups, deploying replicas, encrypting the volumes and the backups, demonstrating to a third party auditor all of the above... and mind that there might be many other things I honestly ignore!
I'm not saying I won't go that path, it might be a good idea after a certain scale, but in the first and second year of a startup your mind should 100% be on "How can I make my customer happy" rather than "We failed again the audit, we won't have the SOC 2 Type I certification in time to sign that new customer".
If deciding between Hetzner and AWS was so easy, one of them might not be pricing its services correctly.
I’m wondering if it makes sense to distribute your architecture so that workers who do most of the heavy lifting are in hetzner, while the other stuff is in costly AWS. On the other hand this means you don’t have easy access to S3, etc.
Depends on how data-heavy the work is. We run a bunch of gpu training jobs on other clouds with the data ending up in S3 - the extra transfer costs wrt what we save on getting the gpus from the cheapest cloud available, it makes a lot of sense.
Also, just availability of these things on AWS has been a real pain - I think every startup got a lot of credits there, so flood of people trying to then use them.
Is it though? This is a genuine question. My intuition is that the investment of time / stress / risk to become good at this is unlikely to have high ROI to either the person putting in that time or to the business paying them to do so. But maybe that's not right.
It's more nuanced for sure than my pithy comment suggests. I've done both self-managed and managed and felt it was a good use of my time to self-manage given the size of the organizations, the profile of the workloads and the cost differential. There is a whole universe of technology businesses that do not earn SV/FAANG levels of ROI - for them, self-managed is a reasonable allocation of effort.
One point to keep in mind is that the effort is not constant. Once you reach a certain level of competency and stability in your setup, there is not much difference in time spent.
I also felt that self-managed gave us more flexibility in terms of tuning.
My final point is that any investment in databases whether as a developer or as an ops person is long-lived and will pay dividends for a longer time than almost all other technologies.
Managing the PostgreSQL databases is a medium to low complexity task as I see it.
Take two equivalent machines, set up with streaming replication exactly as described in the documentation, add Bacula for backups to an off-site location for point-in-time recovery.
We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of.
Add monitoring that the DB servers are working, replication is up-to-date and the backups are working.
> We haven't felt the need to set up auto fail-over to the hot spare; that would take some extra effort (and is included with AWS equivalents?) but nothing I'd be scared of.
this part is actually scariest, since there are like 10 different 3rd party solutions of unknown stability and maintanability.
> Managing the PostgreSQL databases is a medium to low complexity task as I see it.
Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that.
I am a software dev for our small org and I run the servers and services we need. I use ansible and terraform to automate as much as I can. And recently I have added LLMs to the mix. If something goes wrong, I ask Claude to use the ansible and terraform skills that I created for it, to find out what is going on. It is surprisingly good at this. Similarly I use LLMs to create new services or change configuration on existing ones. I review the changes before they are applied, but this process greatly simplifies service management.
> Same here. But, I assume you have managed PostgreSQL in the past. I have. There are a large number of people software devs who have not. For them, it is not a low complexity task. And I can understand that.
I'd say needing to read the documentation for the first time is what bumps it up from low complexity to medium. And then at medium you should still do it if there's a significant cost difference.
You can ask an AI or StackOverflow or whatever for the correct way to start a standby replica, though I think the PostgreSQL documentation is very good.
But if you were in my team I'd expect you to have read at least some of the documentation for any service you provision (self-hosted or cloud) and be able to explain how it is configured, and to document any caveats, surprises or special concerns and where our setup differs / will differ from the documented default. That could be comments in a provisioning file, or in the internal wiki.
That probably increases our baseline complexity since "I pressed a button on AWS YOLO" isn't accepted. I think it increases our reliability and reduces our overall complexity by avoiding a proliferation of services.
I hope this is the correct service, "Amazon RDS for PostgreSQL"? [1]
The main pair of PostgreSQL servers we have at work each have two 32-core (64-vthread) CPUs, so I think that's 128 vCPU each in AWS terms. They also have 768GiB RAM. This is more than we need, and you'll see why at the end, but I'll be generous and leave this as the default the calculator suggests, which is db.m5.12xlarge with 48 vCPU and 192GiB RAM.
That would cost $6559/month, or less if reserved which I assume makes sense in this case — $106400 for 3 years.
Each server has 2TB of RAID disk, of which currently 1TB is used for database data.
That is an additional $245/month.
"CloudWatch Database Insights" looks to be more detailed than the monitoring tool we have, so I will exclude that ($438/month) and exclude the auto-failover proxy as ours is a manual failover.
With the 3-year upfront cost this is $115000, or $192000 for 5 years.
Alternatively, buying two of yesterday's [2] list-price [3] Dell servers which I think are close enough is $40k with five years warranty (including next-business-day replacement parts as necessary).
That leaves $150000 for hosting, which as you can see from [4] won't come anywhere close.
We overprovision the DB server so it has the same CPU and RAM as our processing cluster nodes — that means we can swap things around in some emergency as we can easily handle one fewer cluster node, though this has never been necessary.
When the servers are out of warranty, depending on your business, you may be able to continue using them for a non-prod environment. Significant failures are still very unusual, but minor failures (HDDs) are more common and something we need to know how to handle anyway.
For what it's worth, I have also managed my own databases, but that's exactly why I don't think it's a good use of my time. Because it does take time! And managed database options are abundant, inexpensive, and perform well. So I just don't really see the appeal of putting time into this.
If you have a database, you still have work to do - optimizing, understanding indexes, etc. Managed services don't solve these problems for you magically and once you do them, just running the db itself isn't such a big deal and it's probably easier to tune for what you want to do.
Absolutely yes. But you have to do this either way. So it's just purely additive work to run the infrastructure as well.
I think if it were true that the tuning is easier if you run the infrastructure yourself, then this would be a good point. But in my experience, this isn't the case for a couple reasons. First of all, the majority of tuning wins (indexes, etc.) are not on the infrastructure side, so it's not a big win to run it yourself. But then also, the professionals working at a managed DB vendor are better at doing the kind of tuning that is useful on the infra side.
Maybe, but you're paying through the nose continually for something you could learn to do once - or someone on your team could easily learn with a little time and practice. Like, if this is a tradeoff you want to make, it's fine, but at some point learning that 10% more can halve your hosting costs so it's well worth it.
It's not the learning, it's the ongoing commitment of time and energy into something that is not a differentiator for the business (unless it is actually a database hosting business).
I can see how the cost savings could justify that, but I think it makes sense to bias toward avoiding investing in things that are not related to the core competency of the business.
This sounds medium to high complexity to me. You need to do all those things, and also have multiple people who know how to do them, and also make sure that you don't lose all the people who know how to do them, and have one of those people on call to be able to troubleshoot and fix things if they go wrong, and have processes around all that. (At least if you are running in production with real customers depending on you, you should have all those things.)
With a managed solution, all of that is amortized into your monthly payment, and you're sharing the cost of it across all the customers of the provider of the managed offering.
Personally, I would rather focus on things that are in or at least closer to the core competency of our business, and hire out this kind of thing.
You are right. Are you actually seriously considering whether to go fully managed or self managed at this point? Pls go AWS route and thank me later :)
I ran through roughly our numbers here [1], it looks like self-hosted costs us about 25% of AWS.
I didn't include labour costs, but the self-hosted tasks (set up of hardware, OS, DB, backup, monitoring, replacing a failed component which would be really unusual) are small compared to the labour costs of the DB generally (optimizing indices, moving data around for testing etc, restoring from a backup).
Yes thank you for that. I always feel like these up front cost analyses miss (or underrate) the ongoing operational cost to monitor and be on call to fix infrastructure when problems occur. But I do find it plausible that the cost savings are such that this can be a wise choice nonetheless.
I really do not think so. Most startups should rather focus on their core competency and direct engineering resources to their edge. When you are $100 mln ARR then feel free to mess around with whatever db setup you want.
Or CDN, queues, log service, observability, distributed storage. I am not even sure what the people in the on-prem vs cloud argument think. If you need a highly specialised infra with one or two core services and a lower tier network is ok then on-prem is ok. Otherwise if is a never ending quest to re-discover the millions of engineering hours went into building something like AWS.
Dead on. Recently, 3 and 4 have been compelling. Cloud costs have rocketed up. I started my casual transition to co-lo 2 years ago and just in december finished everything. I have more capacity at about 30% of the cost. If you go option 3, you even get the benefit of 6+ month retro pricing for RAM/storage. I'm running all DDR4, but I have so much of it I don't know what to do with it.
The flip side is that compliance is a little more involved. Rather than, say, carve out a whole swathe of SOC-2 ops, I have to coordinate some controls. It's not a lot, and it's still a lot lighter than I used to do 10+ years ago. Just something to consider.
I don't know. I rent a bare metal server for $500 a month, which is way overkill. It takes almost no time to manage -- maybe a few hours a year -- and can handle almost anything I throw at it. Maybe my needs are too simple though?
I pay around an extra $200/month for "premium" support and Acronis backups, both of which have come in handy, but are probably not necessary. (Automated backups to AWS are actually pretty cheap.) It definitely helps with peace of mind, though.
I have a similar system from Hetzner. I pay around $100 for it. No bandwidth cap.
I have setup encrypted backups to go to my backup server in the office. We have a gigabit service at the office. Critical data changes are backed up every hour and full backup once a day.
Yeah -- I know I could probably get a better deal. I pay more for premium support ($200), as well as a North American location. Plus, probably an addition premium for not wanting to go through the effort of switching servers.
5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).
There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).
Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?
If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?
I think this is one of the more interesting areas to think about, at least for me!
When I worked IT for a school district at the beginning of my career (2006-2007), I was blown away that every school had a MASSIVE server room (my office at each school - the MDF). 3-5 racks filled (depending on school size and connection speed to the central DC - data closet) 50-75% was networking equipment (5 PCs per class hardwired), 10% was the Novell Netware server(s) and storage, the other 15% was application storage for app distributions on login.
Personally I haven't seen a scenario where it makes sense beyond a small experimental lab where you value the ability to tinker physically with the hardware regularly.
Offices are usually very expensive real estate in city centers and with very limited cooling capabilities.
Then again the US is a different place, they don't have cities like in Europe (bar NYC).
If you are a bank or a bookmaker or similar you may well want to have total control of physical access to the machines. I know one bookmaker I worked with had their own mini-datacenter, mainly because of physical security.
I am pretty forward-thinking but even when I started writing my first web server 30+ years ago I didn’t foresee the day when the phrase “my bookie’s datacenter” might cross my lips.
If you have less than a rack of hardware, if you have physical security requirements, and/or your hardware is used in the office more than from the internet, it can make sense.
5 was a great option for ml work last year since colo rented didn't come with a 10kW cable. With ram, sd and GPU prices the way they are now I have no idea what you'd need to do.
Thank goodness we did all the capex before the OpenAI ram deal and expensive nvidia gpus were the worst we had to deal with.
An interesting question, so time for some 100% speculation.
It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost.
So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business."
I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.
Why would a client who wants to run a Blockchain be risky for Herzner? I'm not a fan, I just don't see the issue. If the client pays their monthly bill, who cares if they're using the machine to mine for Bitcoin?
They are certain to run the machines at 100% continually, which will cost more than a typical customer who doesn't do this, and leave the old machines with less second-hand value for their auction thing afterwards.
I’d bet that main reason would be power. Running machines at 100% doesn’t subtract much extra , but a server running hard for 24 hours would use more power than a bursty workload.
Also very subject to wildly unstable market dynamics. If it's profitable to mine, they'll want as much capacity as they can get, leading Hetzner to over provision. Then once it becomes unprofitable, they'll want to stop all mining, leaving a ton of idle, unpaid machines. Better to have stable customers that don't swing 0-100 utilization depending on ability to arbitrage compute costs.
I wouldn't be surprised if mining is also associated with fraud (e.g. using stolen credit cards to buy compute).
Netflix might be spending as much as $120m (but probably a little less), and I thought they were probably Amazon's biggest customer. Does someone (single-buyer) spend more than that with AWS?
Hertzner's revenue is somewhere around $400m, so probably a little scary taking on an additional 30% revenue from a single customer, and Netflix's shareholders would probably be worried about risk relying on a vendor that is much smaller than them.
Sometimes if the companies are friendly to the idea, they could form a joint venture or maybe Netflix could just acquire Hertzner (and compete with Amazon?), but I think it unlikely Hertzner could take on Netflix-sized for nontechnical reasons.
However increasing pop capacity by 30% within 6mo is pretty realistic, so I think they'd probably be able to physically service Netflix without changing too much if management could get comfortable with the idea
A $120M spend on AWS is equivalent to around a $12M spend on Hetzner Dedicated (likely even less, the factor is 10-20x in my experience), so that would be 3% of their revenue from a single customer.
> A $120M spend on AWS is equivalent to around a $12M spend on Hetzner Dedicated (likely even less, the factor is 10-20x in my experience), so that would be 3% of their revenue from a single customer.
I'm not convinced.
I assume someone at Netflix has thought about this, because if that were true and as simple as you say, Netflix would simply just buy Hetzner.
I think there lots of reasons you could have this experience, and it still wouldn't be Netflix's experience.
For one, big applications tend to get discounts. A decade ago when I (the company I was working for) was paying Amazon a mere $0,2M a month and getting much better prices from my account manager than were posted on the website.
There are other reasons (mostly from my own experiences pricing/costing big applications, but also due to some exotic/unusual Amazon features I'm sure Netflix depends on) but this is probably big enough: Volume gets discounts, and at Netflix-size I would expect spectacular discounts.
I do not think we can estimate the factor better than 1.5-2x without a really good example/case-study of a company someplace in-between: How big are the companies you're thinking about? If they're not spending at least $5m a month I doubt the figures would be indicative of the kind of savings Netflix could expect.
We run our own infrastructure, sometimes with our own fincing (4), sometimes external (3). The cost is in tens of millions per year.
When I used to compare to aws, only egress at list price costs as much as my whole infra hosting. All of it.
I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating original content, rather than cloud bill.
> I would be very interested to understand why netflix does not go 3/4 route. I would speculate that they get more return from putting money in optimising costs for creating original content, rather than cloud bill.
I invest in Netflix, which means I'm giving them some fast cash to grow that business.
I'm not giving them cash so that they can have cash.
If they share a business plan that involves them having cash to do X, I wonder why they aren't just taking my cash to do X.
They know this. That's why on the investors calls they don't talk about "optimising costs" unless they're in trouble.
I understand self-hosting and self-building saves money in the long-long term, and so I do this in my own business, but I'm also not a public company constantly raising money.
> When I used to compare to aws, only egress at list price costs as much as my whole infra hosting. All of it.
I'm a mere 0,1% of your spend, and I get discounts.
Of course netflix is optimising costs, otherwise it would not be a business, I just think they put much more effort elsewhere. They could be using other words, like "financial discipline" :)
My point is that even if I get 20 times discount on egress its still nowhere close, since i have to buy everything else - compute, storage are more expensive, and even with 5-10x discounts from list price its not worth it.
(Our cloud bills are in the millions as well, I am familiar with what discounts we can get)
How is this reasonable? At what point do they pull a Dropbox and de-AWS? I can’t think of why they would gain with AWS over in house hosting at that point.
I’m not surprised, but you’d think there would be some point where they would decide to build a data center of their own. It’s a mature enough company.
I'm largely just thinking $HUGE when throwing out that number, but there are plenty of companies that have cloud costs in that range. A quick search brings up Walmart, Meta, Netflix, Spotify, Snap, JP Morgan.
> Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills
back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a person was full stack in a sense that they covered backups, networking, firewalls, and knew how to source hardware.
the argument was that this was too expensive and the cloud was better. so hundreds of thousands of SME's embraced the cloud - most of them never needed Google-type of scale, but got sucked into the "recurring revenue" grift that is SaaS.
If you opposed this mentality you were basically saying "we as a company will never scale this much" which was at best "toxic" and at worst "career-ending".
The thing is these ancient skills still exist. And most orgs simply do not need AWS type of scale. European orgs would do well to revisit these basic ideas. And Hetzner or Lithus would be a much more natural (and honest) fit for these companies.
I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.
I don't think I've ever seen a non-hot-swap disk in a normal server. The oldest I dealt with had 16 HDDs per server, and only 12 were accessible from the outside, bu the 4 internal ones were still hot-swap after taking the cover off.
Even some really old (2000s-era) junk I found in a cupboard at work was all hot-swap drives.
But more realistically in this case, you tell the data centre "remote hands" person that a new HDD will arrive next-day from Dell, and it's to go in server XYZ in rack V-U at drive position T. This may well be a free service, assuming normal failure rates.
Yes, I did write that a bit hasty. I changed above to the normal process. As it happened we just installed a server without hotswap disk, but to be fair that is the first one I have personally seen in the last 20 years.
Remote hands is a thing indeed. Servers also tend to be mostly pre-built now days by server retailers, even when buying more custom made ones like servermicro where you pick each component. There isn't that many parts to a generic server purchase. Its a chassi, motherboard, cpu, memory, and disks. PSU tend to be determined by the motherboard/chassi choice, same with disk backplanes/raid/ipmi/network/cables/ventilation/shrouds. The biggest work is in doing the correct purchase, not in the assembly. Once delivered you put on the rails, install any additional item not pre-built, put it in the rack and plug in the cables.
In the Bay Area there are little datacenters that will happily colocate a rack for you and will even provide an engineer who can swap disks. The service is called “remote hands”. It may still be faster to drive over.
It baffles me that my career trajectory somehow managed to insulate me from ever having to deal with the cloud, while such esoteric skills as swapping a hot swap disk or racking and cabling a new blade chassis are apparently on the order of finding a COBOL developer now. Really?
I can promise you that large financial institutions still have datacenters. Many, many, many datacenters!
we had two racks in our office of mostly developers. If you have an office you already have a rack for switches and patch panels. Adding a few servers is obvious.
Software development isn't a typical SME however. Mike's Fish and Chips will not buy a server and that's fine.
This space of #2 like Lithus is not something I'm very familiar with, so thank you for the comment that piqued my interest!
If you're willing to share, I'm curious who else you would describe as being in this space.
My last decade and a half or so of experience has all been in cloud services, and prior to that it was #3 or #4. What was striking to me when I went to the Lithus website was that I couldn't figure out any details without hitting a "Schedule a Call" button. This makes it difficult for me to map my experiences in using cloud services onto what Lithus offers. Can I use terraform? How does the kubernetes offering work? How does the ML/AI data pipelines work? To me, it would be nice if I could try it out in a very limited way as self-service, or at least read some technical documentation. Without that, I'm left wondering how it works. I'm sure this is a conscious decision to not do this, and for good reasons, but I thought I'd share my impressions!
Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list.
We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around.
What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am.
Your team also has full access to the Kubernetes cluster to deploy to as you wish.
I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.
This makes total sense to me. I'm thinking through the flow that would lead me to be a customer of yours.
In my current job, I think we're honestly a bit past the phase where I would want to take on a migration to a service like yours. We already have a good team of infrastructure folks running our cloud infrastructure, and we have accepted the lock-in of various AWS managed services. So the high-touch devops support doesn't sound that useful to me (we already have people who are good at this), and replacing all the locked-in components seems unlikely to have good ROI. I think we'd be more likely to go straight to #3 if we decided to take that on to save money.
But I'll probably be a founder or early employee at a new startup again someday, and I'm intrigued by your offering from that perspective. But it seems pretty clear to me that I shouldn't call you up on day 1, because I'm going to be nowhere near $5k a month, and I want to move faster than calling someone up to talk about my needs. I want to self-serve a small amount of usage, and cloud services seem really great for that. But this is how they get you! Once you've started with a particular cloud service, it's always easiest to take on more lock-in.
At some point between these two situations, though, I can see where your offering would be great. But the decision point isn't all that clear to me. In my experience, by the time you start looking at your AWS bill and thinking "crap, that seems pretty expensive", you have better things to do than an infrastructure migration, and you have taken on some lock-in.
I do like the idea of high-touch services to solve the breaking-the-lock-in challenge! I'll certainly keep this in mind next time I find myself in this middle ground where the cloud starts feeling more expensive than it's worth, but we don't want to go straight to #3.
Unfortunately, (successful) startups can quickly get trapped into this option. If they're growing fast, everyone on the board will ask why you'd move to another option at the first place. The cloud becomes a very deep local minimum that's hard to get out off.
> 4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.
Is it still the cheapest after you take into account that skills, scale, cap-ex and long term lock-in also have opportunity costs?
Personal experience: Did some cloud stuff for SME, and later on started colocation. I think my learning curve for all the cloud-stuff was the same as for all the colocation stuff, except the cloud will not get you rid of firewalls, NAT, DHCP and all that stuff. Cloud isn't that much easier, it's just a little bit different. IMHO, the largest disadvantage of colocation is that it requires (sometimes) physical presence at a datacenter.
The cloud is great when you just need to start and when you do not know what scale you will need. Minimal initial cost and no wasted time over planning things you do not know enough about.
The cloud is horrible for steady-state demand. You are over-paying for your base load. If your demand does not scale that much, you do not benefit from the flexibility. Distance from the edge can cause performance problems. In an effort to “save money” you will chase complexity and bake in total reliance on your cloud provider. AAA l
Starting with the cloud makes sense. Just make sure not to engineer a solution you cannot take somewhere else.
As you scale and demand becomes known, you can start to migrate some stuff on premises or to other managed providers.
The great thing about “cloud architecture” is that you can use a hybrid model. You can selectively move parts of the stack. You can host your baseline demand and still rely on the cloud for scalability.
Where you need to spend the money and gain the expertise is in design. Not a giant features waterfall but rather knowing how to build an application and infrastructure that is adaptable and portable as you scale.
The core services are cheap. S3 is cheap. Dynamo is cheap. Lambda is exceedingly cheap. Not understanding these services on their own terms and failing to read the documentation can lead one to use them in highly inefficient ways.
The "cloud" isn't just "another type of server." It's another type of /service/. Every costly stack I've seen fails to accept this truth.
Been using Hetzner Cloud for Kubernetes and generally like it, but it has its limitations. The network is highly unpredictable. You at best get 2Gbit/s, but at worst a few hundreds of Mbit/s.
Is that for the virtual private network? I heard some people say that you actually get higher bandwidth if you're using the public network instead of the private network within Hetzner, which is a little bit crazy.
Can someone explain 2 to me. How is a managed private cloud different from full cloud? Like you are still using AWS or Azure but you are keeping all your operation in a bundled, portable way, so you can leave that provider easily at any time, rather than becoming very dependent on them? Is it like staying provider-agnostic but still cloud based?
To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof).
It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team.
This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues.
If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment.
Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do.
Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers.
And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck.
Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers
You give customers root but you're on call when something goes tits up?
You're a brave DevOps team. That would cause a lot of friction in my experience, since people with root or other administrative privileges do naughty things, but others are getting called in on Saturday afternoon.
From a platform risk perspective, each tenant has dedicated resources, so it's their platform to blow up. If a customer with root access blows up their own system, then the resources from the MSP to fix it are billable, and the after-action meetings would likely include a review of whether that access is appropriate, if additional training is needed to prevent those issues in the future (also billable), or if the customer-provider relationship is the right fit. Will the on-call resource be having a bad time fixing someone else's screw up? Yeah, and having been that guy before, I empathize. The business can and should manage this relationship however, so that it doesn't become an undue burden on their support teams. A customer platform that is always getting broken at 4pm on a Friday when an overzealous customer admin is going in and deciding to run arbitrary kubectl commands takes support capacity away from other customers when a major incident happens, regardless of how much you're making in support billing.
This is essentially how it is. Additionally, the reality is that our customers don't often even need to think about using root access, but they have it if they want it. They are putting a lot of trust in us, so we also put trust in them.
Instead of using the Cloud's own Kubernetes service, for example, you just buy the compute and run your own Kubernetes cluster. At a certain scale that is going to be cheaper if you have to know how. And since you are no longer tied to which services are provided and you just need access to compute and storage. you can also shop around for better prices than Amazon or Azure since you can really go to any provider of a VPS.
Getting rid of bureaucratic internal IT department is a game changer for productivity. That alone is worth 10x infra costs, especially for big companies where work can grind to a halt dealing with obstructionists through service now. Good leaders understand this.
Sadly true. Or, the so-called internal IT Dept. can be a shambolic mess of PHB's, Brunchlords, Catberts, metric maximizers, and micromanagers, presiding over the hollowed-out and burned out remains of the actual workforce that you'd need to reliably do the job.
I am using something inbetween 2 and 3, a hosted Web-site and database service with excellent customer support. On shared hardware it is 22 €/month. A managed server on dedicated hardware starts at about 50 €/month.
Where do AWS reserved instances come into your hierarchy? What if there existed a “perpetual” reserved instance? Is cap-ex vs. op-ex really the key distinction?
if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible!
plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have
There are a bunch of ways to manage bare metal servers apart from Nix. People have been doing it for years. Kickstart, theforeman, maas, etc, [0]. Many to choose from according to your needs and layers you want them to manage.
Reality is these days you just boot a basic image that runs containers
Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)
Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.
Agreed. The one bastion of sanity in all this is (/was) the UK. I formed my first company there, 18 years ago, online, in 30 minutes, for around £20.
I then moved to Portugal and started not one but two companies there. The whole process is so clearly setup to discourage people from actually forming companies. Everything from the attitude of all involved ("are you sure?!"), to the practical bureaucracy and costs involved.
I thought perhaps Portugal was just an outlier. But then I moved to Germany. And just wow. Definitely worse. Rounds of paperwork, notary offices, and fees. A process taking weeks. And for a GmbH a minimum investment of €25k. Sure you can form a UG. company for €1, but that effectively just announces "don't trust us, we're tiny" (IMHO).
It is something that really saddens and frustrates me with the EU.
Edit: And sure, you can form an Estonian company. But then you have to try and fly under the radar with regards to the 'permanent establishment' rules.
> And sure, you can form an Estonian company. But then you have to try and fly under the radar with regards to the 'permanent establishment' rules.
It depends on what you’re trying to achieve. If you want to do some tax optimization shenanigans, then yeah, don’t think it’ll work for you (if you’re in the EU, at least). But if you just want to set up a small to mid-sized company without dealing with German bureaucracy, I think Estonia is your best bet right now. I don’t think (IANAL) you would get into trouble for having a permanent establishment – you’ll just have to still deal with the DE tax office. But if EU-INC really can unify the processes for all countries, I’m all for it!
> The whole process is so clearly setup to discourage people from actually forming companies.
By the way, it’s really backwards in Estonia. I’ve seen zero sole proprietors when I was living there – everybody just opens an LLC.
Yes you can form a company in Ireland while living in France. But you cannot get an Irish VAT number without a physical presence in Ireland.
And if – for example – France learns that you are running an Irish company from France (i.e. you have a 'permanent establishment' in France), they'll want you to file and pay French corporation tax. Which is likely sufficiently annoying that you may as well have formed a French company in the first place.
In Portugal (for example) is is mandatory for an Lda (limited company) to have an employee, and they must be paid minimum monthly amount. In practice this means a few hundred euros a month go to the government in taxes. Then on top of that is another hundred or so in accountancy fees.
As someone who had a company in Portugal I do not understand what you meant about requiring an employee to have limited liability company. You may be referring to the need for one of the owners needing to pay social security but it only needs to do so if it does not pay in any other form. It hurts but it is not unreasonable and may provide unemployment benefits.
The accountant requirement I confirm, but also true in Poland for example.
The really big offense is that one must pay taxes for the next year based on the previous tax payments. They give a small grace period but then it’s on. It was supposed to be an emergency measure but of course it was never removed. It is a free loan to the state.
You can open a single person business though and skip the accountant.
We've been using JuiceFS in production for a few months now and I'm a big fan. I've felt for a while that block-level filesystems do not adapt at all well to being implemented across a network (with my personal experience being of AWS EBS and OpenEBS Mayastor). So the fact that JuiceFS is interfaces at the POSIX layer felt intuitively better to me.
I also like that it can keep a local read-cache, rather than having to hit object storage for every read. This is because it can perform a freshness check with with the (relatively fast) metadata store to determine if its cached data is valid, prior to serving the request from cache.
We back it with a 3-node (Redis-compatible) HA Valkey cluster, and in-cluster MinIO object storage, all in bare-metal Kubernetes. We can saturate a 25g NIC with (IIRC) 16+ concurrent users.
It is also one of the few Kubernetes storage providers that provides read-write-many (RWX) access. Which can also be rather helpful in some situations.
In an early test we were running it against MinIO with zero redundancy. Which is not recommended in any case. There we did see some file corruption creep in. In which case some files in JuiceFS become unreadable, but the system as a whole kept working.
Another reason I think JuiceFS works well, is indeed because of its custom block-based storage format. It is disconcerting because you cannot see your files in object storage, but instead just a lot of chunks. But this does buy some real performance benefits, especially when doing partial file reads or updates.
Another test we're doing is running a small-to-medium sized Prometheus persisted to JuiceFS. It hasn't shown any issues so far.
And, if you've made it this, far: check us out if you want a hand installing and operating this kind of infra: https://lithus.eu . We deploy to bare-metal Hetzner.
It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.
We're looking at alternatives, I've made some previous comments on that front. Sadly MinIO was the only option with sufficient performance for this particular situation. Thankfully we're not using any MinIO-specific features, so at least the migration path away is clear.
reply