with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do.

> let different applications use the messages (handy for cross cutting event/notification systems)

rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible.

how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off?

not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.

I know other systems have semi-similar mechanisms, however most of them retain the “someone is the sole owner of this message” style design, which I think is fundamentally limiting. Owning application dies, is it acked or not? Acks but never gets around to putting it back on the queue? Who takes priority if 2 separate applications wish to watch the same stream of events?

I think Kafka’s “nobody owns it, acks are consumer group level” give you the same advantages for the application itself, without a number of the more difficult complications.

> rmq also does a publish once and fanout to multiple queues to support this

Which is probably fine for small volume or velocity topics, but is going to cause all sorts of load issues at higher scale.

each app would get its own queue, the messages would hit a fanout exchange that would route the same message to both queues.

Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.

Kafka had been extremely reliable with latency, even under load in my experience.

If you’ve got badly lagging consumers that are trying to read from very old points in the topic while everyone else is at the head, you’ll definitely see some increased resource usage, but again, that’s mostly a consumer issue, and I’ve need seen performance degrades that much.

yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong)

its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for.

Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it can scale, but also it can just sulk.

rabbit has it's own set of problems, and frankly it's probably not choose either anymore.

Currently I'm using DDS, specifically from eprosma. I would avoid that implementation unless you're using java.

I really like NATS. However I would probably use what every is bundled with the cloud system I'm using, unless its super critical.

MQTT is quite nice for things, as is rabbit.

You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...

^ this.

All they needed was a database, or possibly a DB that supports row signing. I mean actually they could have done it with git. They don't publish that many stories an hour.

Everything about this setup is just plain wrong, and to then boast about it, absolute madness.

Maybe 95% of cases, but not all.

Long message processing time really kills kafka in a way it doesn't kill Rabbit Mq. Combine it with inherent read paralelism being limited to the number of partitions. Add in high variability of message rates and bingo, that's like 90% of the issues I've had with kafka over the years.