You have a problem!
The minimum latency (green) at around 20ms is reasonable. The blue high latency spikes suggest that the connection is being heavily used - which could be down to people catching up from when the connection was down.
Does the modem get switched off at any time? What other electrical.electronic equipment is location close to the modem?
Can you go back over say 10 days and provide a link to each of those - it could show a pattern.
Then, can you get the basic connection stats - Attenuation, SNR and sync speed along with error counts.
M H C
taurus excreta cerebrum vincit