|
|
|
I get full bandwidth from the Fritzbox that zen sent me, but I want more control, data, metrics and insights into my network.
Looking around the net, there is a lot of conflicting info about the ER-X, the microtik hex-s, the erl-3, and more , . specifically as to whether any of these devices can really do full gbit over pppoe whilst applying any of their extended capabilities like smart queuing, etc. And they seem ... old ?
And so I am wondering if it would be a wasted purchase to buy an er-x or erl-3 and find I have to run with all the smarts turned off just to get what I paid for in terms of bandwidth .
Wondering what the current advice is and what people have done, what performance they are getting etc. , basically what am I missing if anything or is this the right product set to be choosing from (erl-3 sounds like the most powerful there?) and will I be disappointed with the performance?
|
|
|
For a gigabit link I doubt the Mikrotik hEX S will cut it with its dual-core 880MHz MIPS processor, but the RB4011 will easily manage, with its quad core 1.4GHz ARM.
I have both the hEX PoE (single core MIPS) and the RB4011.
The worst-case scenario with Mikrotik is when you are routing traffic that can't be handled by FastTrack; that includes all IPv6, and some scenarios which involve VLAN tagging and bridging.
With LAN-to-LAN tests, I found my hEX PoE could only just about manage 300Mbps of simple iperf3 traffic without FastTrack. But the RB4011 can handle about a gigabit on a single core in the same situation, so there's plenty of headroom. And in normal use, IPv4 over PPPoE *is* handled by FastTrack.
I have Netflow enabled, but I don't bother with any sort of queuing. I don't have any Ubiquiti gear to compare with (other than their wireless access points).
|
|
|
|
I've got a couple of ER-Xs to go with FritzBoxes. Let me know if you get them and make them work. I gave up and put them back in their boxes. (Although I was just trying to use them for VPN, not as a FB replacement. I couldn't get them to VPN into a FB.)
I'm fairly sure I did the research and think they are Gigabit. But dunno about stressing them. The ER-Xs are cheap enough, you could always buy a couple and split the work among them!
FWIW, I found them not as easy to set up as the Internet claims. You really do need to know what you are doing. I obviously don't.
|
|
Register (or login) on our website and you will not see this ad.
|
|
|
For data, metrics and inights then the Ubiquiti Dream Machine Pro will give you a lot of what you are after. Will handle fullGbit without issue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
M H C
taurus excreta cerebrum vincit
|
|
|
An ER-X won't do gigabit over pppoe so you can rule that out. It's fine if you're just using it as a switch though. In a LAN it can switch 1gb/s no problems at all but routing traffic is a different beast altogether.
|
|
|
|
I’d go a step further and recommend the OP, if they are looking to “future proof” only consider Mikrotik routers that run ARM mult-core processors rather than any legacy box that has MIPS or Tilera processors.
The ARM processors will just kill them in sheer performance for anything that is unable to scale multi-threaded wise - PPPoE and single tunnel VPN spring to mind.
|
|
|
|
Mikrotik either the RB4011 or the new RB5009UG+S+IN, if you have need for the 2.5G port, although the later is all sold out everywhere for at least the next month.
|
|
|
Got a UDMP here sat in its box since day one as it still cant cope with a 1G pppoe connection, look at their forums its full of people moaning about it. Metrics/DPI last time I tested it were also fairly useless, most things went to unknown on DPI.
Im not bashing, I have £1000's of ubnt kit, but I stick to their switches and AP's.
I still use WFilter by IMFirewall, software that I bought over 10 years ago tells me all I need to know on what and who is using my internet connection - runs on a small PC from mirror port on my Draytek.
|
|
|
|
Good suggestion. Unfortunately shortages and delays on all manner of goods at the moment. It’s causing me several headaches sourcing equipment that keeps getting further and further delayed. If it’s in stock it gets snapped up. We have real supply chain issues at the moment.
|
|
|
|
I was in a similar boat, bought a UDMP and was highly disappointed in it. Had an ERPro-8 for a while, until I wanted to upgrade to something else recently (first choice was a UDMP which I sent back). I liked both the EdgeRouter I had and the EdgeSwitch that I still have. Ultimately went for a MikroTik CCR2004-16G-2S+ after being advised, highly satisfied with it although some users may find it complicated or not within their budget perhaps (most everyday consumers at least won't be looking for such a router).
|
|
|
|
The CCR2004-16G-2S+ is an awesome piece of kit, but the RB4011 is almost as powerful at half the price. The RB4011 doesn't have dual PSU, and it only has a single 10G interface, and its switch can't do VLAN switching in hardware - but those aren't major concerns for home users.
A downside of the CCR2004, at least in the short term, is that it only runs RouterOS v7 which is still in beta/RC phase only (and has been for several years).
|
|
|
For data, metrics and inights then the Ubiquiti Dream Machine Pro will give you a lot of what you are after. Will handle fullGbit without issue
This box and the non pro specifically dont handle PPPOA/E packets at full speed which applies to a huge amount of the UK fibre network unless you are on sky/virgin really. They dont have hardware acceleration for the packets and thus are pegged by the single core limitation to encode them so people are seeing much slower speeds than full GB speeds when using them. There is a HUGE thread on unify about the issues and lack of fix.
They are very nice looking boxes but there are quite a few firmware issues and the PPPOA issue is huge for most people on an openreach based fibre connection. I really hope they get it fixed or release new hardware with hardware offloading.
The older boxes like edgerouters etc have hardware offloading for PPPOA/E so they *should* be full speed.
Some people are trying to do 1:1 NAT with an hardware accelerated box infront of the UDM to try and get around the issue.
Edited by Derpy (Tue 21-Sep-21 16:45:40)
|
|
|
For data, metrics and inights then the Ubiquiti Dream Machine Pro will give you a lot of what you are after. Will handle fullGbit without issue
This box and the non pro specifically dont handle PPPOA/E packets at full speed which applies to a huge amount of the UK fibre network unless you are on sky/virgin really. They dont have hardware acceleration for the packets and thus are pegged by the single core limitation to encode them so people are seeing much slower speeds than full GB speeds when using them. There is a HUGE thread on unify about the issues and lack of fix.
They are very nice looking boxes but there are quite a few firmware issues and the PPPOA issue is huge for most people on an openreach based fibre connection. I really hope they get it fixed or release new hardware with hardware offloading.
The older boxes like edgerouters etc have hardware offloading for PPPOA/E so they *should* be full speed.
Some people are trying to do 1:1 NAT with an hardware accelerated box infront of the UDM to try and get around the issue.
Apparently, according to UI in the past few days they are hopeful of a firmware update to “improve” the situation, they have carefully stopped short of saying that they will fix it. I have heard rumours that the latest UDMP SE may have sorted the issue.
|
|
|
For data, metrics and inights then the Ubiquiti Dream Machine Pro will give you a lot of what you are after. Will handle fullGbit without issue My Qotom running pfsense with an Intel i7 processor cost less those. My network connection uses DHCP not PPPoE so I don't know what difference that makes.
Michael Chare
|
|
|
For data, metrics and inights then the Ubiquiti Dream Machine Pro will give you a lot of what you are after. Will handle fullGbit without issue My Qotom running pfsense with an Intel i7 processor cost less those. My network connection uses DHCP not PPPoE so I don't know what difference that makes.
I'm interested in these.
What's the power consumption like on those Qotom boxes Michael?
I like the idea of an x86 powered router, with options to run pfSense or others...but would like something fairly efficient.
Currently the Linksys WRT1900ACS (OpenWRT) is showing its age a little.
|
|
|
|
x86 boxes are a neat way of doing this too.
If you have a VLAN-capable switch, and are happy to run a "router on a stick" into a single port, then you can get away with something extremely low power like one of the low-end Intel NUCs. Even a rPi 4 would do (but I'm not sure what user-friendly routing firmware to suggest running on that).
Low-power PCs with dual NICs are available, but a bit harder to find.
BTW, pfSense has recently gone commercial: the "community edition" that you can download is not longer the same as the software supplied on their boxes, and will diverge more over time. Maybe time to look at OPNsense.
|
|
|
x86 boxes are a neat way of doing this too.
If you have a VLAN-capable switch, and are happy to run a "router on a stick" into a single port, then you can get away with something extremely low power like one of the low-end Intel NUCs. Even a rPi 4 would do (but I'm not sure what user-friendly routing firmware to suggest running on that).
Low-power PCs with dual NICs are available, but a bit harder to find.
BTW, pfSense has recently gone commercial: the "community edition" that you can download is not longer the same as the software supplied on their boxes, and will diverge more over time. Maybe time to look at OPNsense.
The OpenWRT build for the RPi4 (and 3) seems to work pretty well (I tried it for a short while on the RPi3) just not sure it's powerful enough.
The pi combined with a USB3 Ethernet adapter means a simple setup too.
|
|
|
|
This is interesting
So the RB4011 can easily do Gbit over pppoe IF fast track is enabled - this presumably implies a lot of the extended software driven features are off or very constrained down to bare bones ?
|
|
|
This is interesting
So the RB4011 can easily do Gbit over pppoe IF fast track is enabled - this presumably implies a lot of the extended software driven features are off or very constrained down to bare bones ?
From my testing, the RB4011 can do almost a Gbit in a single stream *without* fast track enabled - that's if you're just talking about forwarding 1500-byte MTU packets (iperf3/TCP). Obviously, smaller packets will tax it more.
In that situation, a single iperf3 stream fills a single core: a typical snapshot looks like this
| Text | 1
23
4 | [admin@gw2] > /system resource monitor
cpu-used: 24% cpu-used-per-cpu: 0%,97%,0%,0%
free-memory: 980656KiB |
(That's iperf3 between two local hosts on two different VLANs, using tagged packets into the RB4011, so it doesn't work with fasttrack)
In any sort of multi-user scenario, you'll be sharing the traffic across cores. Using iperf3 -c x.x.x.x -P 4:
| Text | 1
23
4 | [admin@gw2] > /system resource monitor
cpu-used: 38% cpu-used-per-cpu: 12%,53%,67%,20%
free-memory: 980712KiB |
And if it's IPv4 with or without PPPoE, and doesn't involve ingressing on VLAN interfaces, then you'll get fasttrack, so it'll use a lot less CPU.
I'm confident enough that it will handle a gigabit of real-world traffic.
|
|
|
Apparently, according to UI in the past few days they are hopeful of a firmware update to “improve” the situation, they have carefully stopped short of saying that they will fix it. I have heard rumours that the latest UDMP SE may have sorted the issue.
Thanks I hadnt spotted that, fingers crossed, I have seen some saying to SE fixes the issues and a few saying it doesnt so its still a bit up in the air right now. Hopefully the UDM will get an SE branch as well if its a hardware fix.
Right now though I would not be buying a UDM/UDM Pro if you have a GB PPPOA connection unless you are prepared to 1:1 NAT or have it hit the speed. Otherwise its a decent bit of kit.
Edited by Derpy (Wed 22-Sep-21 00:07:15)
|
|
|
So far we seem to be saying RB4011 (££), maybe a UDM Pro (££££), maybe an RPi 4 with a USB dongle NIC (£), and maybe an intel pfSense box (£££).
What are the metrics and oversight like on the RB4011, I will go and rtm I think before I buy one.
Does anyone have a view on the ERL- line like ERL-3 ; sounds like it might be underpowered alongside the RB4011 still ?
Hmm I have just spotted the dual Gbit ethernet carrier board for a pi 4 compute module, maybe that's a better route forward.
Edited by arthurdent1234 (Wed 22-Sep-21 13:08:24)
|
|
|
I use the CLI only (not their web GUI, and not their "winbox" tool). I get my observability from exporting Netflow into nfdump/nfsen. Old school, in other words
These days, for many networks, I think a mirror port on a switch, feeding into some sniffing/analysis tool (e.g. packetbeat, bro, snort etc) would be the way to go. The advantage here is that it's not inline, so it's not going to impact your traffic performance at all. Packetbeat into elasticsearch would give you some pretty awesome observability.
Much traffic is encrypted these days, but at least you can see source/dest IPs, and (for the time being at least) you can look at TLS SNI to see the hostnames of sites people are connecting to, even if they are encrypted.
|
|
|
|
So maybe I am looking at 2 Rpi4 compute modules + carrier boards, one for O11Y and one for OpenWRT and use a semi managed switch between them ?
Certainly looks like a project that would keep me busy, still wondering if its easier to buy a box and be done with it for now.
|
|
|
|
My advice would be to get some kit and run pfSense on it, you can then upgrade for faster speeds in the future without a learning curve for a new UI and OS, and in many cases transfer over your configuration with a few clicks.
You can either go for an official pfSense box or get something like an i7 Kettop network appliance or similar. I'm running an i7 7500 box at 5 watts idle power consumption, no fans so quiet, got pfBlocker on there handling IP filtering and various firewall rules, and it routes 1Gb PPPoE without breaking a sweat. Numerous help articles available for pfSense and very little it can't do. Also packet capture for debug purposes is built in.
|
|
|
My advice would be to get some kit and run pfSense on it, you can then upgrade for faster speeds in the future without a learning curve for a new UI and OS, and in many cases transfer over your configuration with a few clicks.
You can either go for an official pfSense box or get something like an i7 Kettop network appliance or similar. I'm running an i7 7500 box at 5 watts idle power consumption, no fans so quiet, got pfBlocker on there handling IP filtering and various firewall rules, and it routes 1Gb PPPoE without breaking a sweat. Numerous help articles available for pfSense and very little it can't do. Also packet capture for debug purposes is built in.
Don't go PfSense now, OpnSense is more feature rich, doesn't have updates that break things and isn't run by a very dodgy company. Also PfSense is going to start locking features behind a paywall, which given OpnSense is so good now is a good way to cause your users to migrate to the better competitor.
PfSense used to be good, but for the past 3+ years has been on a downward trajectory and now best avoided.
|
|
|
It is swings and roundabouts really, some people prefer one some prefer the other.
As for updates I've heard the opposite with OPNSense having too regular updates and then bugs to fix, with pfSense going many weeks or months on a stable version between updates.
Essentially they are both a user interface over FreeBSD's firewall, so not so different under the bonent, so each to their own. A good break down of the differences with no agenda is at https://teklager.se/en/pfsense-vs-opnsense so good for those confused between the two, you can't really go wrong with either.
Edited by E300 (Wed 22-Sep-21 15:29:18)
|
|
|
|
This is pushing the cost/benefit curve a bit -- thats £550 of appliance ?
|
|
|
|
You don't need an i7 to route 1Gbps of traffic. i3 or even Celeron would be fine.
If you're going to crank up DPI to the max on the same box then that might be a different story - but as I already said, I think that's better to be kept separate.
|
|
|
|
£300 I paid for a i7 barebones from the Kettop shop, and you don't necessarily need something that fast, they have loads of options. You can also find the same things on aliexpress etc.
|
|
|
My Qotom has a PSU that delivers 12v and up to 5A. I have a meter that shows that the PSU is drawing 8 watts when not doing much. The case does not have a fan but does get warm but not hot to touch.
Michael Chare
|
|
|
This Amazon page for the Qotom made me smile.
It's sold by the "Hot Mini PC Store" - not something you necessarily want to hear for a passively cooled device
|
|
|
|
Be careful because many of those Intel CPU's are self bricking due to the Intel LPC (low pin count) bug.
|
|
|
|
...would've once said buy ARM-based processors and support British semiconductor design greatness ....but not sure who you'd be supporting now; Softbank' Vision fund or Nvidia (if they're ever allowed to complete the purchase).
|
|
|
|
A good point. Hopefully I would think now in 2021 this isn't a problem on new equipment, yes some of the cheaper boxes might be using the affected stepping (some form of Atom C2000) to keep costs low, but it is only an issue if the LPC bus is used. Typically the problem was the bus was used to read the BIOS ROM at boot up, so once it stopped working that was it. However they can just as easily switch to using SPI for reading the BIOS ROM for any newer designs. Even kit from 2016, not all kit was affected as some didn't use the LPC bus at all.
|
|
|
This Amazon page for the Qotom made me smile.
It's sold by the "Hot Mini PC Store" - not something you necessarily want to hear for a passively cooled device 
Go for a dual core i5 over that quad core celeron for pppoe.
Edited by Chrysalis (Fri 24-Sep-21 02:36:40)
|
|
|
A good point. Hopefully I would think now in 2021 this isn't a problem on new equipment, yes some of the cheaper boxes might be using the affected stepping (some form of Atom C2000) to keep costs low, but it is only an issue if the LPC bus is used. Typically the problem was the bus was used to read the BIOS ROM at boot up, so once it stopped working that was it. However they can just as easily switch to using SPI for reading the BIOS ROM for any newer designs. Even kit from 2016, not all kit was affected as some didn't use the LPC bus at all.
Er, no. The LPC degradation and related circuit degradation are wide spread among Intel CPU's to a greater or lesser extent. It first raised it's head with the Atom C2000, but it is present in many other CPU's.
The LPC, USB and SD Card buses circuitry degradation issues also apply to other Bay Trail processors such as Intel Celeron J1900 and N2800/N2900 series.[21] and also to Pentium N3500, J2850, J2900 series and Celeron J1800 and J1750 series as those are based on the same affected silicon. I believe the E3800 Atoms are also effected.
I suggest you do some research on the issue and you will find most of those cheap mini PC's are based on effected silicon. I know I am sitting on a timebomb of a J1900 motherboard that I didn't find out till after I purchased it
|
|
|
|
Yeah, I fell foul of this with my QNAP NAS.
As @jabuzzard says, it's worth being aware of and checking.
|
|
|
The LPC, USB and SD Card buses circuitry degradation issues also apply to other Bay Trail processors such as Intel Celeron J1900 and N2800/N2900 series.[21] and also to Pentium N3500, J2850, J2900 series and Celeron J1800 and J1750 series as those are based on the same affected silicon. I believe the E3800 Atoms are also effected.
The above looks like a quote from https://en.wikipedia.org/wiki/Silvermont - in which case reference [21] is https://www.intel.com/content/dam/www/public/us/en/d...
That refers to certain steppings of J1800, J1900, N2807 and N2930 only.
Not all N2800 series are affected. I have nine DN2820 NUCs in service for over 7 years (they make great Perfsonar nodes) and none has failed.
|
|
|
|
Yes it is a cut and paste because it was a quick way of showing that it is much more than the C2000 series Avaton Atom's that are effected. Even that list from Wikipedia is far from comprehensive and not all CPU's have been given new stepping's to fix the problem.
I have just applied a firmware patch to our Omnipath switches at work which are E3800 based that mitigates the issue. It does not say it explicitly, but the erratum has a line about minimising the USB usage to prolong switch life or some such thing. There is only one reason to do that.... From memory they where well north of £20k each. Not remotely funny.
I am pretty sure a bunch of our 10/40Gb network switches are impacted too. Fortunately all logging has always been off switch to a syslog host so the onboard USB is not heavily used.
It is a complete sh!t show and that is far from played out. It is at least as bad a the bad capacitor debacle. The difference it is much easier to recap something than replace a soldered on BGA CPU.
|
|
|
|
Very early indications seem to suggest they have fixed it, I guess it’s a case of watch this space!!
|
|
|
|
And exactly how in advance when making a purchase do you know if you are getting a corrected stepping? You can't so it's pot luck. Further not all CPU's are getting an updated stepping either. As I said it's a complete sh!t show on Intel's behalf.
|