Yep adding noise would mess up the deltas which would reduce the efficiency. This can be mitigated with a quantizer, which I was hoping to write about next.

I suspect it's possible to achieve 99% lossy compression without getting too complex to be understandable in the toy codec with a couple more of these similar techniques.