Switch Latency - nothing you would notice, almost certainly well under 1mS. My Gbit switch add tens of microseconds.
A patch panel is 8, 12,16 , 24, or 48 RJ45 modules in a single mounting and are designed to manage the cables too. And a patch panel will be a lot cheaper than using RJ 45 modules.
Use back boxes and modules at the other end where you have 1 or 2 sockets.
It will give you three advantages (at least) and a small 8 port switch is under £20
1. More ports available - just connect one link from router to switch
2. The switch does the work rather than put extra processing onto the hub - if a hub is heavily loaded managing the DSL/WAN interface, DHCP server, wireless interface, and the switching function, it can impact on the response times for interrupts and thus latency.
3. The switch can give you (providing you use all 4 pairs) 1Gbit networking which hubs tend not to do.
You can continue to use the hub for wireless and it will all look like one small network and if you want you could patch your games console direct to a port on the hub.
The cleanest way is 2 cat5e cables to each location and those brought back to a patch panel. Use short patch leads from panel to an 8 port switch. If you want to increase in future, just add another switch to the hub.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
M H C
taurus excreta cerebrum vincit