From the article "Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead." and from your comment "We do the same for the CDN and always use canary testing if possible. We unfortunately never assumed this piece of software could cause all the DNS servers to stack overflow."
This read like the DNS software is being changed. As some people already mentioned is this a corruption where checksum would of been prevented the stack overflow or would a canary detected this? Why would a change to DNS server software not canaried?
I read it as "DNS software changed, that worked fine, but it turns out we sometimes generate a broken database - not often enough to see it hit during canary, but devastating when it finally happened"
GP also notes that this database changed perhaps every 30 seconds
Just a few guesses.. if you have a process that corrupts a random byte every 100.000 runs, and you run it every 30 seconds, it might take days before you're at 50% odds of having seen it happening. and if that used to be a text or JSON database, flipping a random bit might not even corrupt anything important. Or if the code swallows the exception at some level, it might even self-heal after 30 seconds when new data comes in, causing an unnoticed blib in the monitoring if at all
Now I don't know what binary pack does exactly, but if you were to replace the above process with something that compresses data, a flipped bit will corrupt a lot more data, often everything from that point forwards (where text or json is pretty self-syncronizing). And if your new code falls over completely if that happens, no more self-healing.
I can totally imagine missing an event like that during canary testing
Have you looked at Cuelang [1] as a configuration language or source of truth?. It's still new but it's a superset of JSON but attempted to combine data/schema into one and reject inheritance and replace with validation instead.
XDP UDP mapping to firecracker vms via WireGuard is really interesting! I have a question a bit before UDP is landed on the NIC, assuming the NICs on the edge servers is connected to multiple transit providers for incoming and outgoing traffic. This mean from the VM perspective, you can have incoming/outgoing tap/tun inside the VMs able to receive packets from difference transits or outbound, did you do anything with this aspect? and if so do you also deal with ECMP inbound in such that you can have the same virtual IP receiving UDP on multiple edge servers?
I also want to point out that Manuel Schoelling been working on and released a p2p [1] distributed hash table [2] work on top of WireGuard to build a VPN mesh. He talked about this recently [3] at FOSSEM.
Choptix without the need for choptix rest. My friend have been using this choptix for a litter over a year and decided to make it available for anyone else who want it.
