Huge Ping changes around 9pm / later evening

For some reason later in the evening my connection Ping jumps from a 6-9ish in the daytime, to 500 - 600 around 9ish.

This changes itself back to the lower figure in the morning.

My only way of fixing it in the evening is unplugging my modem, and plugging it in again... but it takes a lot longer to connect than normal.

Anyone else have this problem?