I don’t know if you actually read the article or not but just to be clear that was not the argument that was made.
It’s not just “not new” it’s considered to be actively worse than all of its peers in terms of things like performance and complexity, it’s actively a very questionable choice in a greenfield project in 2023.
I agree with you, there was definitely a fixation. The implication is that React doesn't do things the new and better way just because it's old, rather than a deliberate decision.
Every time this topic comes up, it seems people invariably divide into one of two groups. In the first group are those who trust in the odds and play the probability game. In the other, those who demand absolute guarantees. Neither group seems fully capable of understanding the other's standpoint.
Let me add my own perspective: Distributed locks are a fallacy. They can be beneficial in decreasing contention, under the assumption that "most of the time, only one actor will be active". However, by themselves, they offer no solid guarantees. The blog post addresses this point by introducing the concept of fencing tokens. These tokens have the potential to provide concrete guarantees, but they require the cooperation of downstream systems for enforcement, which isn't always possible.
I was really surprised to see Antirez argue for the probability approach.
I don't know why you say neither group understands each other when I literally said in my comment that my approach is probabillistic.
Also, how is configuring an infinite timeout (and checking whether the client is actually gone before manually breaking the lock lease) not an absolute guarantee?
Okay but then I really don't know what point you're trying to make. Either an infinite timeout, or non-absolute mutual exclusion guarantee: we have to pick either one. The right choice depends on the workload. There is no perfect solution that lets you have your cake and eat it too.
A use case that I wish authorization service providers would talk more about is support for "list" queries e.g. What resources of type foo can the user read?
In really simple cases you may model this as a one-shot check on a logical collection resource e.g. If I have an organization, and an organization owns many repositories, I may check for action:read on resource:/organizations/:id/repositories. It's very limiting though. What if I want to list all repositories across all organizations I have access to? What if I have multiple levels I want to cut across? Do I need to do all the sub-queries and aggregations myself? Do I need to do a one-shot check on every potential resource? What if the answer is "no" for every one, and I end up doing a table scan of the entire DB just to produce an empty result set? etc.
The problem you're describing is sometimes called the "ACL Aware Indexing" problem [1]. In a permissions service based on Zanzibar, this can partially be solved by walking the permissions graph in a reverse fashion: SpiceDB exposes this API as the LookupResource API [2]. However, even this approach has performance implications for very large datasets, so we're also working on improving resource lookup via a LookupWatch API [3], which can be used by consumers to actively cache the available set of resources as access changes over time.
According to one of the original designers of Zanzibar, they specifically designed the semantics to be able to answer this [1]. Whether a given implementation is able to do this efficiently is another matter though!
You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.
Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.
edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.
Can't resist: Overheard in Stanford locker room after Show HN. "He got me" L said of D's dunk over him. That fking D boomed me. L added "He's so good", repeating it four times. L then said he wanted to add D to the list of players he works out with this summer.
Numerous AWS (really early) internal eng talks I saw, hammered home the point that distributed locking / leasing was one of the hardest problems to engineer for, if not the hardest, often quoting Chubby and Paxos Made Live papers.
Today, if I were to need a form of distributed co-ordination, I'd reach for Durable Objects to see if it fits my needs.
The Python client implementation can be improved, for sure. In particular, the pathological case that is difficult to deal with is the one where the heartbeat thread pauses at the worst possible time (the "pathological hang" you mention) and so the main thread doesn't notice its lease has expired.
Lockable is designed as an easy-to-use drop-in locking mechanism which requires minimal changes client-side. The downside is that this puts the onus on the client implementation to fully cover all pathological cases.
I agree that the documentation should be clearer about these limitations and requirements on the client.
If I understood correctly, the problem you're describing is essentially the following. The locks have a timeout to them to avoid zombie locks etc. However this means we can have system A obtain a lock (or a lease, if you prefer) and begin a long-running process on the given resource. However, during the long-running process, the lock times out and system B can acquire a lock, thinking it's the only one using the resource, leading to a conflict. Did I understand you correctly?
We once had a bug from this incorrect use of "distributed locks". A server we accessed under a lock suddenly started lagging past the timeout of the lock, another server using the lock assumed the lock was released (i.e. timed out) and acquired it, while the original server assumed it still owned the lock. Data corruption occurred.
This implementation has "heartbeats" so I wonder whether it solves the problem.
In order to deal with long-running processes, the client Python implementation uses a separate thread for sending periodic heartbeats to the lockable server, which serves to do 2 things:
1. renew the lease so it doesn't expire which would release the lock
2. notify the main worker thread in the even the lock has been lost
The GP's point was that the heartbeat thread can hang in pathological cases, which means the main worker thread would not be notified that it has lost the lock.
This can be addressed in a few ways - one way being by adding fencing tokens[0]. However, that requires modifying the underlying resource you are accessing.
The user could create software that thinks a resource is locked under their exclusive control, but is not. This can lead to data corruption, which could lead to even errors propagating throughout a system.
There's so much more to CI/CD than the build definitions (e.g. dashboarding, log viewing/retention, access control, manual interventions, secret management, test reporting, etc.) and while some of your points resonate very strongly with me (e.g. local builds), I can't help but wonder what the endgame is here?
You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?
Without more transparency its easy to imagine something like...
Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in"
Step 2. Introduce your own CI solution, which people can now easily switch to
Step 3. Lock people in
Good question. No, we don't intend to replace the CI service. We think of CI as infrastructure, and there are plenty of great infrastructure providers out there. And you are absolutely correct that for Dagger to succeed, it has to remain agnostic to infrastructure providers, which means we cannot build an infrastructure provider business ourselves. If the experience of hosting Dagger becomes so bad that it affects the developer experience, we might ship such a feature as a stopgap and even charge for it - but it is not a strategic priority. The sooner other infrastructure providers offer a fantastic hosting experience, the better. That is not where we see the biggest opportunity for Dagger.
However, there is a great opportunity to help businesses manage their software supply chain. What is running where, and how did it get there? Who is authorized to run which pipelines on which machines, and which which inputs? What is the diff between this dev environment and staging? Staging and production? Etc. Keep in mind this is not limited to your production pipeline; there are staging environments, development environments, all running pipelines all day long. It's hard to simply know everything that is going on.
Each time Dagger runs your pipeline, it can produce telemetry about every aspect of your supply chain. Git SHAs, docker image checksums and cryptographic signatures, specific versions of kubernetes and cloudformation configurations, before and after templating, test results, monitoring status... It also integrates with your secrets management backends, your developer's local source code. Basically every node in your supply chain can be a node in your DAG if you want it to. The logical next step is to give you a centralized place to send all this telemetry, and tools to extract insights from it. You could also perhaps manage the configuration of your various Dagger engines in one location.
Another product that is often requested is a visual DAG debugger. When a pipeline break, you want to know why, and staring at your CI logs is definitely not the best experience for that. With a web UI, there's a lot we can do there.
The business opportunity boils down to this: if CI/CD pipelines are software, there ought to be a platform for developing that software, and an ecosystem of developers creating value around that platform. Dagger aspires to create that missing developer ecosystem. If we succeed, there will be no shortage of business opportunities. If we fail, none of the other features will matter.
question coming from a sales guy among this crowd...
Who does this impact the most and what outcomes does that help them achieve? At the end of the day, you need to pay the bills, for you and your team, who is going to sign the dotted line and tell you that yes "I need to invest $100k on this because it solves a major pain!"?
Whilst I can see this as a nice to have, I'm having a hard time understanding who your target market is going to be...
At the end of the day, I presume this is going to end up becoming a full fledged company, with its sales team, marketing and what not. What are you thinking about in terms of revenue channels so far ?
As someone who does a bunch of CI/CD work, the answer is: supply chain and the security therein is a major focus of enterprises right now. Security folks as well as ops folks are keen on this. Big question is can those two groups convince the developers that this is a good idea (though vice versa should be fairly easy at a certain company size).
That's fine, I don't deny that, companies like circleCI and co are doing really well, but Docker is also critical to a lot of enterprise as well and yet the business model was not as strong as initially thought.
I'm curious to understand what the pricing is going to be and whether it's going to be well recieved or not (and right now, there is nowhere on dagger.io's website where pricing is mentioned)
> Each time Dagger runs your pipeline, it can produce telemetry about every aspect of your supply chain.
To me, that still doesn’t seem to go to the core of what value it brings to table. I understand that Dagger can do all this, and that businesses would like to know what runs where and how everything interacts, but… it doesn’t explain to me the CI / build pipeline angle?
How does knowing the telemetry around my build pipelines translate to better software, or cost savings, or other improvements?
If there’s a registry to exchange “components” of a build pipeline, what does that bring me what a regular python / java / etc package can’t?
Don’t get me wrong, I think there are plenty of problems with CI (I have a ~200 job CI pipeline on fire in front of me), I just can’t seem to connect the dots here. :)
> there is a great opportunity to help businesses manage their software supply chain
Yes, very much. There are so many layers, components, and their intricate relations that goes totally ignored today at least in most places. Because, doing so is insane amounts of work. Only BigCos can afford to have dedicated teams for 's/w supply chain management', considering the cost-parity-with-returns. However, the solution on this end that works for BigCo doesn't necessarily work for SMEs & startups. That gap isn't small, if am right.
> Another product that is often requested is a visual DAG debugger. When a pipeline break, you want to know why, and staring at your CI logs is definitely not the best experience for that. With a web UI, there's a lot we can do there.
Yes. This definitely helps. But more than a viz DAG element, people look for an early-warning of a failure. Most common build-failure reasons (other than failed tests) -> expired creds used somewhere in the pipeline, provisioning failed/time-out, problem at some other dependent module totally outside org's control (some OSS/dep). People seem to be bothered equally about how to squash'em rather than just where to squash. Locating the part where pipeline broke is just half the part. Actionable insights as to how that pipeline can be healed is the hard part. And considering the diversity of the ecosystem, that's gonna be a wild ride.
BTW, are you folks hiring? "DevOps OS for enterprises" seems very very enthralling, esp for an old toolmaker.
IMO the best open source infra lock-in strategy is kube
took a while for E/AKS to catch up with G, things like ingress still vary in DX across clouds
'the same everywhere but it only works here'
(no comment on morality of this, quality of kube generally, or whether this team will do the same. docker IMO missed the chance to be a cloud host or a standard interface)
Completely off-topic, but k8s actually removes a lot of the vendor lock-in. Imagine having to migrate a k8s based app from AWS to Azure vs a pure AWS-based setup, provisioned with cloudformation. Sure it'll involve some work, but I know what I'd pick. I've personally ran workloads originally intended to be deployed on GKE on on-prem openshift clusters with very few changes, other than indeed the mentioned ingress stuff.
That was also the initial goals of k8s - make it universal so it's adopted as a standard by everyone else to make yourself competitive against the AWS juggernaut. And it worked.
concern: like every system that we think of as cloud software, the 'cloud platform integration' half of it never gets open sourced
same as like, aws extending mysql and postgres to provide RDS features -- they don't open source those pieces, so 'stock oss' DBs don't have fancy RDS features like backup + restore
with kube, platform-integrated components like network, ingress + storage 1) require custom work on each cloud, so new features won't be available on all clouds at the same time.
But also 2) that means some components, ingress specifically, have different interfaces on different clouds when they do finally get implemented.
> You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?
Assuming that dagger does take over the world, this has the docker story of "this is a widely used tool that a bunch of people are using and we can't monetise without resorting to lock-in", which is a huge bummer.
I am yet to work on a team with a simple, reliable and repeatable build process.
Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.
I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.
Agreed - I’ve spent months working on a deep tech problem before I could even show a functional core, let alone putting it into production. Some problems take time, experience and foresight to solve and the solution only dovetails at the end.
“Someone that avoids talking about what they did, and focuses too much on "we"”
This is very interesting. I naturally use “we” when discussing past products and achievements. I do this in recognition of the fact that things are very rarely designed and built in true isolation. Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward.
I agree. Maybe it's a cultural thing, but if someone talked about a big multiyear project only in terms of I, I would see that as a red flag. Either they actually did all the work themselves in isolation, but that must be because they are hard to work with, or they are taking all the credit for work done by a quite a few people in a team.
If you get asked, multiple times, how exactly you yourself contributed to the project and you continue only talking about what "we" did, it is absolutely a red flag.
> "Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward."
Principal engineers know that there are appropriate times to say "we did..." and appropriate times to lean heavily into saying "I specifically did..." (while crediting others for their part, of course, but as briefly as possible) and they also know which one an interview is.
"We" may not be a red flag (I appreciate a measure of humility and recognition of teamwork as well) but it must always followed up by a question about the details of what the interviewee did.
It depends on the context - but each time I've interviewed someone, the "we" is important, but the "I" is important-er. It's almost implicit that everyone works in a team to achieve big goals, but the reason I'm talking to that candidate is to hear what they did as part of that team as that's what makes a difference to the hiring.
This is one of those things that's getting lost in the intention. I've been in an interview where I used we and the interviewer was fairly blunt in asksing which things I directly did, which I collaborated on, which I reviewed, etc. It becomes clear very quickly they aren't trying to figure out if you are sociable, but to understand what you do and do not know how to do. Shifting into that mindset it becomes easy to say things like "three person team. We collaborated on x,y,z. A and B were my primary responsibilties and I think my most important contributions were blah blah".
Yeah for me as well, it's just a way of recognising teamwork. If anything I would say it's the opposite, excessive "I..me..myself" can become a red flag in some cases.
Speaking as someone who hires, "I" is a huge red flag for me — it's a dead ringer for what is (in a very soft sense), a confidence man. Someone who's aggressively and artificially trying to project an external appearance of competence. I've had multiple bad hires like this, and it's now something I really watch out for.
The worst thing with guys like this isn't that they're going to deliver less than promised, but it's that the aggressive "willingness to lie for their own benefit" bleeds into a lot of other behavior at work.
I prefer using “we” but early on in my career was rejected at an interview because they thought I hadn’t done anything myself - now I mix in we and “I specifically implemented/designed/etc.”
Same. As a long time team lead/manager I always take the blame when things go wrong and give credit to individuals other than myself when things go right.
Someone constantly using "I" like they worked in a vacuum, and didn't play nice with their team is a bigger red flag for me.
The important piece in the interview is can they answer in detail the rest of the questions.
I should have been clear - I start my interview by asking the candidate to focus on their experience and every time they talk about what "we" did on a project I follow up with what specifically they did.
The imposters/charlatans (and weeding them out was the context for this whole thread) find a way to avoid the question.
>"Tell me about the most complicated projects you contributed a significant part of the design, implementation to, or ideally both."
Start with the problem they aimed to solve.
Probe for how they chose the solution, and what alternatives they considered.
Get into the details of specifically what artifacts they produced, and if part of a team what role they played.
Get them to describe the solution in technical detail.
Probe into trade-offs they had to make or compromises.
Ask what aspects they found novel or innovative.
Once they finish describing the system, ask if it was successful, and how they measured it.
If they were around after launch, how did they operate it, monitor it, what unexpected challenges they encountered that they didn't foresee.
By the way, all along it doesn't matter if the domain or technology they are describing to you is totally different from your own experience. A senior+ engineer should be able to explain their domain to someone equally technical in a different domain with some competency.
This verbatim can very well apply to many different industrial engineering efforts, regardless of whether there is software involved.
> I naturally use “we” when discussing past products and achievements
Then you will not get recognition about what you did in the eyes of others. It is hard to trust someone who hides behind a group, because it looks like you have no confidence in yourself or what you did.
Recognition and trust have never been a problem for me. Have you hired at this level before, or merely assuming? As always there is nuance, as other commenters have touched on.
I have hired and been hired at this level and I have tried both approaches, one is IME far superior to the other. Any role which requires self-confidence is one where I would screen for candidates that are comfortable with claiming achievement.
My first experience with event sourcing was having a new hire at a previous company try to shill it. It was a smaller company and he had previously built a career consulting on ES. Every problem we had he would come up with an argument for why ES would solve it. I soon learned he had zero technical ability, but was great at sounding credible to management. I liked to imagine him as a "seagull ES" practitioner. He flies in, sh*ts ES on everything, and flies out without having to deal with the mess.
Ironically I went on to use a lot of ES prinicpals in building a distributed deduplication engine for my previous startup after I left that company. We used a WAL which we were able to rebuild databases/projections from, ship to replica nodes for read replicas, push to S3 for resilient storage etc. it was an extremely powerful architecture for us. It was also complex and required a special skill set to work on and reason about.
I have to imagine anyone that advocates for "moving from CRUD to Event Sourcing" has recently been sucked into drinking the coolaid, and has no idea about the world of hurt they've signed up for, completely necessarily.
Similar thing happened on a project I joined. Except this person not only forced ES onto the business but their own specific library for it https://github.com/johnbywater/eventsourcing
The business eventually failed to due to this, due to slow implementation of simple features and many other issues with it.
I will never use ES due to this project, it's pointless, anything you can do with it, you can do without it.
For the benefit for readers, of course I didn't force event sourcing on anybody.
The above allegation is utter bollocks, which apparently continues a stalking campaign by an ex-colleague who was forced to leave the company by the management.
This also happened FIVE YEARS ago, so the obsessiveness is disturbing.
