Have you ever explored the idea of shaped notes?

There's multiple different approaches with both 4-shape and 7-shape systems being common. But the point is that your color system seems largely correlated to it, and there has been research done on the shape note system.

https://en.wikipedia.org/wiki/Shape_note