I presume office is busy from around 9:30am to 6:45pm
Basic latency looks high, unless you are in Northern Ireland or Scotland.
The issue looks likely to be the ADSL line is dropping connection, or is experiencing lots of errors at times. Have you done the checks where you unplug everything from the phone line and test the ADSL modem in the test socket.
Problem could be as simple as the line has a fax machine attached that is not filtered, or needs a second filter on it.