Exactly. My ping times vary from one day to the next. Double, half, double again, quadruple despite taking the same route through the network (and using fast path). I doubt that's a unique case and would render this system of geolocation pretty useless.
add to that the varying routing involved. eg. someone further away from london than me can have better pings to london simply because their routing is more direct. Take these examples of my last few isp's. to bbc.co.uk.
BTw on fast path, varied from about 17ms up to about 30ms.
ukonline llu fast path, usually 9-10ms occasionally would increase a bit by a few ms.
xilo fast path, over 20ms.
VM cable, has been as low as 11ms, but they have a habit of routing me via leeds which bumps it to 16-17ms.