Reason I say to put VOIP direct to the router is that is is effectively separate from the rest of your network and does not have any local traffic.
As for POE there are different adapters out there. If you use one which injects power onto te line, it should/could only present data and no power to the switch . Does the patch panel need to be? Not really, you are only using low voltage but just choose appropriately.
I am about to install a couple of VOIP base stations here, they can use POE but as there is real power close by there is no need.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
M H C
taurus excreta cerebrum vincit