Doesn't the router in this case form a PPPOE tunnel to the modem? if that has the same IP as a machine on your LAN, how does the router know which interface to use? (IE. LAN or WAN ethernet interface)
No.
The pppoe connection sends ethernet packets between the router and the modem, but these are not IP packets and have no IP address. All the router need to know is which interface to use, not which IP address to connect to: e.g in my rouer I have "plugin pppoe eth1" in the ppp configuration, no need to mention an IP address as the interface name ("eth1") is the only information needed (this is for an ADSL router in bridge mode, but pppoe is the same for everybody!). The only IP address involved will be the one obtained while setting up the PPP session, and that has nothing to do with any IP address assigned to the modem itself - I suppose you could refer to it as a PPP tunnel to the ISP using PPPoE - all the modem is doing is encoding/decoding packets to/from the VDSL link, without even looking inside the packets (*).
Of course, if you mean that you need to access the modem for configuration/statistics, this will need a static route to the modem - otherwise the router will send IP packets which you mean for the modem itself over PPP to the ISP and that won't really work. What will happen in this case is that you bave both pppoe packets (headed in the direction of your ISP) and IP packets (headed to the modem) on the same interface. This will cause no confict.
(*) "without even looking inside the packets"... with the possible exception of a qos setting which will need to look inside the packets to decide how to deal with them, but that's irrelevant and I'm only mentioning it before some other pedant does.