That’s not normally the case. The hub 3 in modem mode only supply’s the one IP, the routable ipv4.

No idea about business service over cable.

You’re over thinking this as far as I can tell. You have your VM hub in modem mode plugged into a switch, and this switch then connects to the LAN side of two OPNsense instances - one physical, one virtual. You then wake up the physical machine if the VM stops responding.

Firstly, I’ve run OPNsense and it doesn’t tend to crash, so there’s a problem somewhere. Secondly, is there a reason you aren’t running OPNsense in HA mode? You get virtual interfaces with virtual MAC addresses and the VM hub wouldn’t see a change.

Thats right it does only supply one IP, but unlike the SH2, SH3 will supply the IP on any of it's four ports.

When either of the OPN machines are powered down the is no activity between the SH and whatever OPN machine is currently off, no NIC lights nothing so the SH3 only seems one machine at a time.

It will not issue the WAN IP to the newly powered on machine without a reboot itself. This ridiculous tbh as I know other folk with VDSL modems that have no issue with this,

There's no over thinking at all really. All I want the thing to do is provide a WAN IP to the newly powered machine when the other is off, as I explained in another reply, SH3 will only ever see one device at a time. There is no NIC traffic between the SH3 and whatever machine is powered off no lights, nothing.

If I boot up the previously powered machine it gets a WAN IP instantly on any of the SH3 ports.

The workflow is, Virtual and Physical both attached directly to the SH3, no switch in the middle. I did try having a switch in the middle, didn't work, SH3 wouldn't given a WAN IP to any of the machines.

Maintenance scenarios, loss of power on the VM host, UPS power running out on VM host, Physical machine has it's on UPS which will last much longer. Its not just for crashing.

I can't run OPNsense in HA mode as it requires 3 WAN IPs which only Virgin provide on a business package, also only with the Hitron in router mode, if I switch to modem you loose the bank of IPs as they're provided by a GRE tunnel. Basically HA is never going to be an option with Virgin plus the Physical machine is just drawing power doing nothing.

I don't understand how the SH3 is seeing the physical machine differently when the WAN NIC MAC address on the Virtual has been spoofed using the same model of PCI card and the exact same system settings.

Yes, I have checked and the Hitron is the same.

Thanks
Dan

I don't understand how the SH3 is seeing the physical machine differently when the WAN NIC MAC address on the Virtual has been spoofed using the same model of PCI card and the exact same system settings

Reads like the issue is at layer 2 as said. What box is between the SH2 and the VM and bare metal box?

No switch between the SH3 and the machines. Both are directly connected.

So if it’s in modem mode, then it’s communicating with the first active device and ignoring the others. Effectively the the other ports cease to exist, at least as far as operation as a switch. Which is understandable. You then lose that active device, but the SH3 isn’t looking for any other active devices on any other ports. So it fails.

Have you tried connecting the devices via an intermediate switch, such that they both connect to the same port on the SH3?

In reply to a post by Pheasant:

Reads like the issue is at layer 2 as said. What box is between the SH2 and the VM and bare metal box?

I'm guessing the bare metal MAC is stopping the virtual machine's generated MAC from being picked up as the single CPE.

The virtual platform is appearing as if it was a switch with two devices attached, then plugged into the Hub.

When your VM crashes it's presumably holding the link up though, so the physical box booting then tries to come up with the same MAC and it's causing issues. Or does your VM crash by powering off?

There's some discussion here about using CARP with DHCP WAN and script examples which might be helpful:
https://forum.opnsense.org/index.php?topic=20972.0

I think trying to connect each machine to the LAN ports on the SH3 might be causing your issues here and would recommend an intermediate switch, especially as one of your symptoms is that you don't even get a link light.