I had my concerns about this as well, so I made a post
about it. You should have a read of the responses, as it links to many others that are very helpful.
But the TL;DR version: It's a priority for your own
connection, not everyone. It's designed to prevent more important services (like streaming, VoIP, etc.) from being degraded by less important ones (like P2P). If all your connection is doing is P2P, priority won't make any difference to the maximum speed you can attain and you can max out your connection providing the P2P quality allows it (I have experienced this, also).
Once you understand the traffic management, you realise it's a good thing.