Colour just groups the latency tests, each bar is one latency test, so yellow is when the speed test was doing nothing else, i.e. idle, green is a test during the download phase and red during the upload phase.
Hover over each of the bars to see the value.
You want particularly during the idle phase for all the latency tests to be similar or same value, during download and upload you get more variation, and the size of the difference between idle and download/upload phases lets you determine something called buffer bloat, i.e. if you connection is very busy in one direction how delayed are small packets or in real world use if doing a massive download and your latency shoots up to 800ms for example then ACKs and other packets linked to opening a web page will be slow to travel back and forth.