The fibre is shared with others, so would be affecting them all, not likely to be congestion on that bit of fibre as multiple GigE links are available.
What is your computers OS? Some may need the RWIN tweaking to avoid stalls.
A screenshot of the graph would be interesting, as sort of proves a point I keep making but seems to be ignored that speedtest.net favours peak speeds.
If there was a major problem with the copper to the cabinet, you'd be expected pings to show some pattern too, rather than a steady value.