|
|
|
Hello Guys
Okay so I'll explain this a little bit. I have an OPNsense Virtual Machine and an HP T610 SFF (acting as a backup) with OPNsense all with the exact same NICs and settings. I'm on the 350/35 tier with a Super Hub 3. Additionally I have a small script running on a Raspberry PI that pings the VM every five mins, if no response is received it sends a WOL packet to the SFF booting it up. I work offshore, and so if my server with it's VMs crashes, I need a working connection to restart them.
Now apparently the Super Hub 3 has a setting built in called "Max-CPE" which means it will only pass the WAN IP to 1 MAC address and one only unless the SH3 is rebooted. This presents a problem for me as when the SFF boots. even though the VM is offline, the SH3 will not pass the WAN IP via DHCP. Even with the NIC MAC on the VM being spoofed on the SFF. This isn't any use as I have to manual reset the SH3.
I don't think there's any solution to this issue with the SH3 and so I was looking at the VOOM 3 business package. Does the Hitron router exhibit the same behaviour with "Max-CPE"?
Also the 4g backup dongle for £7 is quite attractive but can it be plugged into any device or are you forced into attaching it to the Hitron? Does the Hitron have to be in router mode too as i'll be running it in Modem mode?
|
|
|
|
Sorry but I cannot be of any help regarding the equipment you are using, however are you over thinking this? I have a PC that I use for my CCTV which is on 24/7, but every now and then it will lock up, so all I did was use a Kasa smartplug onto the PC, so as long as my home wifi stays up, then I can remotely turn off the plug and turn in back on (BIOS is set to switch PC back on if power available).
|
|
|
|
I have considered using the Pi to power cycle a smart plug, tbh that is going over kill. The SH3 should just issue the WAN IP without rebooting. It's just another example of junk tech from Virgin tbh.
|
|
Register (or login) on our website and you will not see this ad.
|
|
|
|
I use VM in a business setting and they always recommend rebooting the modem first then the router afterwards.
Thanks
Dan
|
|
|
As soon as you have modem mode you are limited to one CPE (ie MAC address) which needs to then do your network management.
It sounds as if your virtual router is the issue, not the single MAC limit.
Some people report issues with the GRE tunnels on the business service; so some searching.
22 years of broadband connectivity since 1999 trial - Live BQM
|
|
|
The VM isn't the issue, it's the SFF bare metal machine. When the Pi detects the VM has went offline after 5 mins it sends the WOL packet which boots up the SFF. At this time there's no router on the network. Logically the SH3 should then allow the SFF to be the router as the VMs NIC is dead.
It doesn't without a power cycle.
If I boot the VM up again, WAN IP no problem.
Edited by mrjay84 (Tue 22-Mar-22 20:09:50)
|
|
|
My guess is the delay in booting the VM means the hub gets some ARP’s from the physical hardware. Not sure any virtualisation platform is really suitable for this usage. You’re going to hit the same problem.
22 years of broadband connectivity since 1999 trial - Live BQM
|
|
|
I've tried two psychical machines in the same config. SH3 simply refuses to give out the WAN IP.
More on this here https://community.virginmedia.com/t5/QuickStart-set-...
I wonder if a static IP is the way to go as then there's nothing to the SH3's DHCP client to hand out?
|
|
|
Nothing to do with IP (layer 3) this is at layer 2.
If you have multiple devices on a switch plugged into the Hub 3 and the Hub 3 is in modem mode you will get this problem. The hub is detecting the first layer 2 device attached via port 1.
Only option is one device, direct Ethernet from port 1 on hub to the device. Any other configuration is going to be intermittent.
You can’t do failover networking with the hub 3 in modem mode.
In router mode you can have as many devices connected as you like.
22 years of broadband connectivity since 1999 trial - Live BQM
Edited by jchamier (Tue 22-Mar-22 21:06:34)
|
|
|
|
Granted, both machines are able to get a WAN IP from all ports on the SH3, I've tested this out, It just needs a power cycle to pick up the other machine as theres no NIC connection when either is off.
Anyway...is the Hitron the same?
|
|
|
That’s not normally the case. The hub 3 in modem mode only supply’s the one IP, the routable ipv4.
No idea about business service over cable.
22 years of broadband connectivity since 1999 trial - Live BQM
|
|
|
|
You’re over thinking this as far as I can tell. You have your VM hub in modem mode plugged into a switch, and this switch then connects to the LAN side of two OPNsense instances - one physical, one virtual. You then wake up the physical machine if the VM stops responding.
Firstly, I’ve run OPNsense and it doesn’t tend to crash, so there’s a problem somewhere. Secondly, is there a reason you aren’t running OPNsense in HA mode? You get virtual interfaces with virtual MAC addresses and the VM hub wouldn’t see a change.
|
|
|
|
Thats right it does only supply one IP, but unlike the SH2, SH3 will supply the IP on any of it's four ports.
When either of the OPN machines are powered down the is no activity between the SH and whatever OPN machine is currently off, no NIC lights nothing so the SH3 only seems one machine at a time.
It will not issue the WAN IP to the newly powered on machine without a reboot itself. This ridiculous tbh as I know other folk with VDSL modems that have no issue with this,
|
|
|
|
There's no over thinking at all really. All I want the thing to do is provide a WAN IP to the newly powered machine when the other is off, as I explained in another reply, SH3 will only ever see one device at a time. There is no NIC traffic between the SH3 and whatever machine is powered off no lights, nothing.
If I boot up the previously powered machine it gets a WAN IP instantly on any of the SH3 ports.
The workflow is, Virtual and Physical both attached directly to the SH3, no switch in the middle. I did try having a switch in the middle, didn't work, SH3 wouldn't given a WAN IP to any of the machines.
Maintenance scenarios, loss of power on the VM host, UPS power running out on VM host, Physical machine has it's on UPS which will last much longer. Its not just for crashing.
I can't run OPNsense in HA mode as it requires 3 WAN IPs which only Virgin provide on a business package, also only with the Hitron in router mode, if I switch to modem you loose the bank of IPs as they're provided by a GRE tunnel. Basically HA is never going to be an option with Virgin plus the Physical machine is just drawing power doing nothing.
I don't understand how the SH3 is seeing the physical machine differently when the WAN NIC MAC address on the Virtual has been spoofed using the same model of PCI card and the exact same system settings.
|
|
|
|
Yes, I have checked and the Hitron is the same.
Thanks
Dan
|
|
|
I don't understand how the SH3 is seeing the physical machine differently when the WAN NIC MAC address on the Virtual has been spoofed using the same model of PCI card and the exact same system settings
Reads like the issue is at layer 2 as said. What box is between the SH2 and the VM and bare metal box?
|
|
|
|
No switch between the SH3 and the machines. Both are directly connected.
|
|
|
|
So if it’s in modem mode, then it’s communicating with the first active device and ignoring the others. Effectively the the other ports cease to exist, at least as far as operation as a switch. Which is understandable. You then lose that active device, but the SH3 isn’t looking for any other active devices on any other ports. So it fails.
Have you tried connecting the devices via an intermediate switch, such that they both connect to the same port on the SH3?
|
|
|
Reads like the issue is at layer 2 as said. What box is between the SH2 and the VM and bare metal box? I'm guessing the bare metal MAC is stopping the virtual machine's generated MAC from being picked up as the single CPE.
The virtual platform is appearing as if it was a switch with two devices attached, then plugged into the Hub.
22 years of broadband connectivity since 1999 trial - Live BQM
|
|
|
When your VM crashes it's presumably holding the link up though, so the physical box booting then tries to come up with the same MAC and it's causing issues. Or does your VM crash by powering off?
There's some discussion here about using CARP with DHCP WAN and script examples which might be helpful:
https://forum.opnsense.org/index.php?topic=20972.0
I think trying to connect each machine to the LAN ports on the SH3 might be causing your issues here and would recommend an intermediate switch, especially as one of your symptoms is that you don't even get a link light.
|
|
|
|
Think I’d next test with two physical machines, hot/cold, with identical MAC etc but both connected via an intermediate switch onto the same port on the SH3
|
|
|
|
Agreed. Think we’re coming to the same conclusion. It’s the either the ports on the SH3 in modem mode being ignored or the the active link isn’t being fully released by the active NIC or there is some MAC caching or other going on by the switch chip in the SH3.
Devices connected via Intermediate switch in to the same SH3 port needed to prove or disprove this theory.
|
|
|
|
Okays thanks Dan. I guess I'll just have to use one machine then with no backup or ditch Virgin media. I do wonder if a static IP would work as I could just enter that into both machines, then the SH3 doesn't have to assign one.
|
|
|
|
Yip. With an intermediate switch neither machine gets an IP
|
|
|
|
Simulated crash by powering off, removing the physical cable so there's definitely not link active.
Unfortunately using a switch results in no IP given to any machine.
|
|
|
|
Just checking that it's a dumb switch, not one that would be trying to get an IP for a management interface
|
|
|
|
It was a dumb switch yeah. I also tried it on a managed one with and without a VLAN.
|
|
|
Unfortunately using a switch results in no IP given to any machine. I think in this scenario the Hub 3 gets two devices sending ARPs and it sees both at the same time and doesn't use either.
The Hub 3 is not designed for this, the assumption is you are connecting a computer or a physical hardware router to the Hub.
It will have a lot of bugs in the firmware, its from Virgin Media !!
22 years of broadband connectivity since 1999 trial - Live BQM
|
|
|
It was a dumb switch yeah. I also tried it on a managed one with and without a VLAN.
Did you define a new VLAN that wasn't shared with the default / management interface of the switch (typically VLAN 1) and then set the ports for the SH3 and the 'routers' to untag for that new VLAN and not be members of any other or default VLAN?
|
|
|
|
The Hub 3 in modem mode will "lock" to the MAC address of the connected machine. If you power off this machince and power on another one, the first MAC address will still be locked to the Hub 3.
The Hub 3 needs to be powered off when the machines are swapped. It's the same advise we give to users on the VM Forums when either changing from Router Mode to Modem mode, or replacing a third party Router.
If you can power off the Hub in-between swapping the machines it should work.
|
|
|
|
One machine is completely off with no power to it's NICs the other is on, it can't possibly see both. WOL packet is sent to the SFF only after the VM shutdown after 5 minutes.
|
|
|
|
I've attempted a Port based VLAN, a tag based VLAN all with unique IDs and a completely dumb swtich.
|
|
|
|
Even when I spoof the VMs WAN MAC to the SFF it still won't issue the WAN IP. This was know in the first post though.
The only thing I can think of it the SH3 locks the MAC AND the Port the MAC was detected on. If this is the case then using a intermediate switch would (in theory) cure this along with the VMs WAN MAC being spoofed one only power to one machine at a time, It's doesn't. As stated in other posts a dumb switch/PVLAN & Tag based VLAN results in no IP again even with only one machine running.
The next thing to try is to spoof the MAC and physically move the ethernet cable over the the Port it appears to have locked on to.
Seen as there's no SSH/Telnet access to the SH3 I need to manually power cycle it. This makes the whole project pointless.
Another solution is to purchase a smart plug where I can get the Pi script to power cycle that when it detects the VM as offline. Of course this plug would have to be LAN only as there's no WAN connection.
|
|
|
Another solution is to purchase a smart plug where I can get the Pi script to power cycle that when it detects the VM as offline. Of course this plug would have to be LAN only as there's no WAN connection.
Isn’t that a different failure scenario to the one which you were attempting to resolve with a primary and backup pfSense instance though?
Even if you had managed to get the above working, you would still have the possibility that the SH3 was unresponsive rather than your devices - so would still need something like this to reset the SH3?
|
|
|
|
Seen as the SH3 won't give up the WAN IP with a reboot it will have to be part of the backup solution otherwise having a backup firewall auto boot is pointless.
Yes that is a possibility, it's remote though as for all the bad features the SH3 has I haven't had to reboot it in years until recently.
|
|
|
Well chaps...
I decided to reinstall OPNsense on the SFF and spoof the WAN MAC (igb1) from the VM, shut the VM down and allowed the SFF to boot from WOL. It got a WAN IP!
To make sure this wasn't a fluke I shut down the SFF and booted the VM, still gets a WAN IP!
I then turned the VM off again and allowed the Pi to boot SFF after 5 mins...again success! I tried this three times now and every time it's worked...Yay!
Now worth noting. Both VM & SFF are plugged into Port 2 & 4 respectively on the SH3, proof the SH3 will give out a WAN IP on any of the ports. Only one is ever powered on at any one time, the other is completely dead, no NIC lights at all. As long as the MAC that the SH3 has locked on to is spoofed it works.
Okay so the next stage in the project is to make script that will SSH into the SFF and tell it to power off when the VM runs, should be easy. If not I can do that part manually.
Thanks to all for responding!
Last question on the original post is anyone knows. The 4G backup dongle that Virgin advertise, can that be used on any device or is it tied into the SH in router mode?
Edited by mrjay84 (Thu 24-Mar-22 16:44:42)
|
|
|
Good news on the OPNsense front.
I wonder why it didn't work previously when you tried with two other SFF/bare metal machines? Weird - did you change anything?
What's the failure scenario that your trying to mitigate with the active / standby OPNsense boxes? May help to understand then if the script idea will be workable in the situation where a OPNsense box becomes non-responsive or OPNsense itself etc? Still a bit unclear to me.
Edit: typo me using pfSense in lieu of OPNsense. Doh!
Edited by Pheasant (Thu 24-Mar-22 18:53:26)
|
|
|
|
No idea mate lol. I did copy and paste the MAC some maybe I messed up somewhere.
The OPNsense VM is hosted on an Unraid server. Sometimes I need to reboot it or power down for maintenance reasons, and the occasional crash. If the VM is off then there's no internet connection, and as I work away half the year, I'm unable to reset it via IMPI without a working connection.
This is where the script comes in. It detects the VM as being offline then boots the SFF which provides an internet connection. Then I can login to servers IMPI and tell it to reboot.
Once Unraid starts the VM it will send a power off command to the SFF telling it to shutdown.
|
Pages in this thread: 
Print Thread
