This is a problem we, as a company, have thought about a lot, but we always concluded that Kubernetes is already the simplest abstraction of a distributed system that is feasible for the diverse needs that the biggest companies out there have.
We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.
With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.
So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.
Same, I've tried three or four times to make it work, including one attempt that just translated compose.yaml into k8s yaml, and every time I came away thinking, "just use k8s". K8s yaml looks complex, and can start to feel very boilerplate, but attempts to hide the complexity often just lead to something not-flexible-enough because it encodes convention over configuration, and inevitably some project runs into limitations and pretty soon you've just built an abstraction layer that leaks or is equally complex/verbose and now you have to learn something new.
Just use k8s and follow similar patterns is the conclusion I've arrived at personally.
To be honest I never really understood the benefit of Docker (Compose) Secrets - which is different from Swarm Secrets. Imho there just plain host mounted volumes, which are hidden from inspect commands?
GP is talking about podman with generated systemd unit files (a.k.a. podman quadlet[0]), not the docker-compose-compatible podman-compose ...and I'd agree, systemd can manage services on a system just fine, and even better than any compose workload ever could.
journald will help with logs, and the pull policy[1] helps with mutable tags. What help do you need with "orphan containers"?
I'm not OP, but the whole podman compose topic gets quite confusing, as initially Podman didn't seem to know what they were trying to do. I've given some more context around it in previous comments.
You shouldn't be using podman compose. It's flimsy and doesn't work very well (at least it was last time I used it prior to Podman v3), and I'm pretty sure it doesn't have Red Hat's direct support.
Instead, activate Podman's Docker API compatibility socket, and simply set your `DOCKER_HOST` env var to that socket, and from there you can use your general docker client commands such as `docker`, `docker compose` and anything else that uses the Docker API. There are very few things that don't work with this, and the few things that don't are advanced setups.
For what it's worth, podman has also a thin wrapper (podman compose) which executes `docker-compose` or the old `podman-compose`. The docs should explain which it picks.
Note:
- `podman-compose` is an early attempt at remaking `docker-compose` v1 but for Podman. This used parsed the compose config and converts them to podman commands, and executes it.
- Later Podman wrote a Docker compatible socket instead, which can work with most docker clis that accept a `DOCKER_HOST` argument, including `docker` and `docker-compose` (both v1 and v2)
- `podman compose` is a thin wrapper that automatically selects `docker-compose` or `podman-compose` depending on which is installed.
Generally all you need is podman, docker-compose (the v2 binary), and that's it. From there you can use `podman` and/or `podman compose`.
One of the nastiest aspects of migrating from docker to podman really is "what to do about docker compose?" coz there are three wildly divergent ways to answer that all of which really suck under certain specific circumstances.
Im no fan of docker and podman by itself is a step up but orchestration headaches are enough to ruin that.
This is what stopped me from picking up Podman more, all our devs use Docker and have been writing compose files for years now. When the response at the time was "you're using Podman wrong, Quadlets are the hot stuff now" it just felt like too big a risk and commitment to jump to at the time. Have things settled more? Getting away from Docker is a bigger priority nowadays for us.
Docker (Compose) has some quirks compared to Podman (Compose), e.g. when using gvisor or a lot of internal networks. Depending on what you do your milage will vary, though.
Agreed. I found compose overlay files merged list values differently between the Docker and Podman versions, which was a PITA in teams running Docker & Linux dev machines.
Then you learn podman can't even list containers for all users properly and it kind of starts smelling like the whole ip4 vs ip6 debacle: bunch of vocal proponents wanting you to subject yourself to endless torture for no discernible reason.
Congrats on building your custom delivery platform. We understand your pain ;-).
Any specific pain you want to share or feature you are still missing in your solution?
We’re happy to help you migrate to Distr if you decide you no longer want to maintain your solution or if you’d like to benefit from regular updates and continuous improvements.
Distr SaaS is configured for USD only (via Stripe) for simplicity, but EUR payments are definitely possible (either via Stripe, SEPA, etc.).
Its all bambi legs at the moment.. I wonder if we'll ever get past this stage.
Moving it all to a 'real' product is an option before it gets too serious. I'll bring this up with the group and lets see what happens.
Thanks for sticking around! Yes indeed, Distr already has a lot of features and can replace many other tools in your deployment stack (e.g., CI, log collection, alerts, licensing, secrets, OCI registry, customer & user management).
Users definitely also use us for internal deployments, but there are some trade-offs you’ll need to make compared to the zero-config setup you get when deploying your Next.js app to Vercel on the one hand, and the hardcore customization you get when running your own GitOps/Terraform setup on the other.
Is there any specific feature you’re missing, or is it just that our website doesn’t really highlight that use case?
Sure, happy to follow up with a detailed comparison.
TL;DR: Octopus Deploy has a strong focus on CD, providing a Cloud based framework to push your software to multiple targets. Distr also supports directly and continuously deploying your software to various deployment targets, but it additionally supports the pull approach, where the customer fully self-manages their infrastructure and simply fetches the application or artifacts from Distr.
Octopus Deploy also offers deep Argo CD integration (after they acquired Codefresh). On the other hand, Distr is completely open source and can be self-hosted if desired.
If you’re interested in specific features, I’m happy to go into more detail.
I feel you, but a huge percentage of recently funded companies are in the AI space. Software distribution for them is even more complex due to all the moving parts, and we want to make sure these companies know that our solution is a great fit for them.
Sure, multiple of our customers that distribute applications with a machine learning/AI component also need to distribute their models. They can use our OCI registry to distribute large images with huge layers. We specifically reworked our registry implementation to storing in-transit blobs on disk to save memory, ensuring the application doesn’t run out of memory [1].
Is registry OOM protection the only advantage your registry has for large layers? Robotics has a need for Docker tooling that handles large layers/images gracefully. Even if you've done the "right" thing and sideloaded your ML models with some other management system, CUDA layers and such are gigantic.
Edit: looking at this, this is very adjacent to some problems w/ robotics deployments. Fleet management, edge deployment, key management. Neat.
I'd be curious about the multi-artifact support. Can I declare a manifest that binds together multiple services (or a service and an ML model?) Do you support ML models as an artifact?
Feel free to reach out via our website[1]. Distr does not require an internet connection to keep your application running. Update commands are fetched directly from the agent and do not require any special connectivity.
Updates are pulled before the rollover to a new version is performed, so a poor internet connection may only affect the download speed of new updates. Distr is designed to operate even when no connection is available, or when connectivity is only allowed in short time slots.
We previously built a package manager for Kubernetes to abstract it in the simplest way possible `glasskube install app` but we failed because every abstraction needs to follow a "convention over configuration" pattern at some point. Also, we weren't able to monetize a package manager.
With Distr (https://github.com/distr-sh/distr), we have actually been able to help companies not only package but distribute and either manage or give their customers a way to self-manage applications. Our customers are able to land on-premises contracts at enterprises way faster than before, which is also a clear ROI for paying for Distr.
So, I don't think that you can get the flexibility of a distributed application orchestrator with a simple declarative YAML file if your target environments are diverse.
reply