This is an excellent idea... but I'm a little put off by the reference implementation using DC/OS and Kafka... Its a little bit 'heavy' for a reference implementation, at least that's my personal preferences.
Id like to see a more 'low tech' version, or in the very least, a plan to transition from a 'heavy' reference implementation to a 'lighter' one once one became available since having a concise reference implementation makes porting and compatibility significantly easier.
Author here. Thanks for the feedback and I agree, a lighter ref impl would be ideal (it's just that I happen to use DC/OS on a daily basis so for me this was the most natural thing to do)
I'm all open for GitHub issues that capture this request and/or PR ;)
The name doesn't really work in this project's favor.
UNIX pipes are stream interfaces, whereas this looks to be message based - that's a fundamental difference.
Named pipes are uniquely identified in a well-defined local namespace, i.e the filesystem, whereas this seems to be an abstraction on top of Kafka with service discovery TBD.
AF_UNIX on linux supports SOCK_DGRAM and SOCK_SEQPACKET in addition to SOCK_STREAM, so having message based interfaces won't be much of a change for some users.
SOCK_DGRAM at least appears to be widely supported (FreeBSD & Illumos & OpenBSD). SOCK_SEQPACKET is at least supported by OpenBSD (I'm going by what's in various docs/man pages).
Generally when a protocol is referred to as message-based it's because there's some framing around the payload. This can make it impractical as compared to a streaming protocol that just passes data back in forth. In some real-time cases, you can't tolerate the latency of waiting to bundle multiple items in a single payload but you also can't tolerate the overhead of framing many small messages.
The most interesting thing I got from this was the DC/OS link, I hadn't heard of it before but I think I'll look at it to see how it might fit with our future direction with/instead of Kubernetes. https://dcos.io/
Not really. Each message has to be an atomic item, not a stream of bytes, since there can be multiple readers consuming a single queue.
The interface specification has problems. "A pull does not remove a message from a dnpipes, it merely delivers its content to the consumer." So if the same consumer pulls the same dnpipe again, does it get the same message? Do messages ever get removed from dnpipes without a reset? Unclear.
Does pull block, support async completions, or just return an error when no data is available?
Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
> ... if the same consumer pulls the same dnpipe again, does it get the same message
No. Each message is delivered at most once. But good point, need to make that clearer!
> Do messages ever get removed from dnpipes without a reset.
No. Again, something I need to clarify as it seems.
> Does pull block, support async completions, or just return an error when no data is available?
It blocks.
> Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
I don't follow. Reset empties the underlying queue.
In general, since the consumers start to consume not from the beginning of time (as in Kafka's `--from-beginning`) but wherever they happen to be (that is, in Kafka terminology from `latest` index) this shouldn't be a problem.
I tried to model as close as possible and as it makes sense after the semantics of (local) named pipes. I might have fudged up here but I'm not 100% clear on where :)
> Do messages ever get removed from dnpipes without a reset.
No. Again, something I need to clarify as it seems.
So what happens after the system has been running for a while? Why doesn't the dnpipe system fill up with old messages that will never be read again? If new subscribers don't see them, and old subscribers have read them, why are they not removed? It would seem that once every subscriber who subscribed before a message was sent has received that message, the message is dead and can be removed. Why keep the history? Did you really mean that?
Also, what happens if one of many subscribers stops making PULL requests? Maybe it's blocked on something, or hung. Do the queues start to build up? Does PUSH eventually block?
> Reset, rather than sending an EOF which passes through the queue, implies that shutdown is either drastic or requires external coordination to empty the queue and stop the sending end before the reset.
I don't follow. Reset empties the underlying queue.
On most queuing systems, when a publisher wants to shut down, they close the channel's sending end or send and EOF message. When all subscribers have read up to the EOF, they close their receiving end. The last messages get processed, and then the subscribers stop. If the only shutdown mechanism is a reset, that can lose messages not yet received. That's OK if the intent is just "kill everything and terminate", but not if you need a clean shutdown.
(ROS, the Robot Operating System, has a publish/subscribe system something like this. It's a soft real time system, so old data is discarded and you never want to block a publisher. They chose to lose messages if a subscriber isn't reading often enough. That's appropriate to a robotics use case. It probably wouldn't be for a containerized web backend. See [1].)
Joking aside: since I've used Kafka for the reference implementation it naturally looks very similar, yes but it's really about the semantics. For example, does your 'normal' Kafka consumer start from the beginning or use latest index, etc.
Id like to see a more 'low tech' version, or in the very least, a plan to transition from a 'heavy' reference implementation to a 'lighter' one once one became available since having a concise reference implementation makes porting and compatibility significantly easier.