I'm way more concerned at how simple it is to introduce a tap on to an analogue phone line. 90% of them have an junction box on the outside of the property secured in some cases by a screw removable without tools.
As a project in middle school (up to age 13 to avoid doubt) I made a phone tap that only triggered when the phone was active and transmitted the call contents via FM all on a device that comfortably fitted inside most of the junction boxes I encountered. Electronics have become smaller and more easily concealable now... and I was a schoolboy hobbyist, not a Taiwanese electronic surveillance specialist.
I guess if someone in the VOIP world wanted to do the same they'd need to send a copy of the packet streams that makes up your call to a third destination without interrupting your call.
As mentioned the pops and clicks would be more indicative of low quality bandwidth between you and the other called party. Depending on your ISP, it could be that they actively rate limit such data to all destinations apart from their own VOIP service or that they generally don't provide enough bandwidth at all. Of course your VOIP provider could also be oversubscribed so lacking in bandwidth.
If you can tell us who your ISP and VOIP providers are then we'd be able to give a more definite answer I'm sure.