One thing of note in the graph is the tracking of response size. This would be very useful for 200 responses with "Error" in the text. Because then the response size would drop drastically below a normal successful response payload size.
In addition to Latency, Error Rates, Throughput and Saturation , folks like Brendan Gregg @ Netflix have recommended tracking capacity.
we are all learning from the same folks ahead of us it seems :)
I agree with other comments though the devil is in the details of how to actually setup these "golden signals" so that they are useful and not just drown everyone in packet level non-sense.
TCP retransmission rates looks like a useful metric which can help in monitoring the health of a service. One way to obtain that is by analyzing service interactions as mentioned in the blog. Tracing could be another way through which we can find that info. I am curious as to how code instrumented monitoring solutions get that information.
(PS: I work for Netsil)
If you look at your latency histogram and are seeing a bump at around 200ms above normal (which was the default minimum wait time a few years back anyway), it's probably retransmits.
Right now we maintain few select percentiles from the latency distribution over 1 min time-period. We plan to maintain latency histograms which will allow you to look at latency distribution on arbitrary time intervals.
Netsil AOC is priced by the number of vCPUs or cores that you would be monitoring. You can reach out to us at hello@netsil.com for the exact price quote based on your needs.
