I used to have a big difference between x1 and x6 speeds; something like 20Mbps/70Mbps (I forget the details now). I was connected PC->gigabit switch->HH3 gigabit port. The problem turned out to be an oddity in compatibility between the switch and the HH3 gigabit port. For some reason, that did not impact most real life browsing/downloading or most tests, but did hit the TTB x1 test.
When I connected the gigabit switch to a 100Mbps port on the HH3, all tests ran at full speed and there were no downsides I can see.
You could be seeing a similar issue, for example if the ethernet ports or drivers on the different machines are causing some mismatch with your router.
Recently moved from BT Infinity 2 to PlusNet. Very happy so far.