I don't believe you can set downstream QOS on your router. It's too late after they have been sent.
It is not perfect, but you can make a shared connection with bulk traffic and gaming work - I started doing this back when everyone had 288/576 sync with BTs BRAS buffer around 600ms long. That was hard.
As downstream bit rates increase it gets easier. It's not free, as I said initially, you have to sacrifice bandwidth.
On a fast ish connection like the OPs you could probably set limits say 85 -90% for normal downloads, for a heavily seeded torrent you would need less.
It is of course not the same as upstream as your line limits the incoming speed, plus you don't have total control, but as you set the QOS to less than downspeed you will build up a queue to work with. The idea then is to let through/prio what you deem latency sensitive and to queue up the bulk with a shortish buffer so you soon drop a packet (head drop is possible with some qdiscs and is preferred). This causes tcp congestion control of the sender to back off and keeps the remote buffer from getting filled up with bulk packets that block your interactive packets.