General Discussion
  >> Fibre Broadband


Register (or login) on our website and you will not see this ad.


Pages in this thread: 1 | 2 | 3 | 4 | (show all)   Print Thread
Standard User mikehiow
(committed) Tue 27-Apr-21 08:26:55
Print Post

Router hardware capable of routing 900mbps over PPPoE (FTTP)


[link to this post]
 
I've hit a bit of a wall with this one.

I've beeing using a J1900 based firewall appliance with pfSense for a while and it's been great until I moved from VM to Zen over FTTP. It would cap out around 400-500mbps and suffer heavy packet loss as it hits the limit.

Based on a recommendation, I picked up an PC Engines APU2 to replace it and that's worse.

It seems there's a particular issue with PPPoE and Intel nics in pfSense, but ultimately the hardware doesn't seem powerful enough to do PPPoE at these speeds anyway, as it the J1900 still "only" does 500-800mbps with Linux based ipfire.

I haven't tried a Linux based solution on the APU2, but I suspect it'll be a similar story.

I see 3 options;
1. Try OpenWRT which will probably run on anything
2. Get better suited, more powerful hardware to keep running pfSense.
3. Find an off the shelf appliance

The trouble is, I can't find too much information on proven solutions.

Goodbye Hyperoptic. Now VM200 frown
Standard User Pheasant
(experienced) Tue 27-Apr-21 09:00:35
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
PPPoE can be quite resource intensive, especially at the top end of speeds now possible with FTTP etc and if you're relying on hardware that is based on desktop-class processors that have to do all the heavy lifting (together with routing, packet inspection etc) they can feel the pinch.

Dedicated appliances or routers often or not get around this by offloading this task or have dedicated capacity in silicon do do the computation, rather than a general purpose CPU, so they generally don't tend to suffer as much as x86 based solutions - insofar as they have sufficient routing throughput - you can still cap out a weedy router.

Edited by Pheasant (Tue 27-Apr-21 09:11:50)

Standard User brookheather
(member) Tue 27-Apr-21 09:20:01
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
I use an HP 600 G2 SFF with Core i3-6100 for pfSense and it works very well - you can buy them second hand for under £100. The 600 and 800 SFF models come with a Platinum rated PSU so power consumption is only around 10W.

Cerberus FTTP + pfSense + UniFi nanoHD


Register (or login) on our website and you will not see this ad.

Standard User E300
(member) Tue 27-Apr-21 09:35:08
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
I found the same as you, the APU2 wouldn't do more than around 300-400 Meg for me. It is because PPPoE isn't multithreaded on pfSense/FreeBSD, so you need a fast enough CPU on one core to get the throughput. This might be fixed in FreeBSD in the future but it's been an issue for a long time so I wouldn't hold my breath.

I ended up getting a i7 7500U, search for Qotom i7 7500u or Kettop i7 7500u, they come fanless in a solid heavy heaksinked case with 6 Intel Ethernet ports. I'd recommend getting one bare bones and putting in your own RAM (8Gig is plenty) and a small mini-SSD, if bought with these already bundled you take a chance on the brand of memory and SSD. They do sell them on Amazon, you have 2 choices, UK stock or direct from China. UK stock, expect to pay around £200 extra for the privilege I got mine direct from China in about a week, it's been great.

An i7 7500U is probably overkill even for a 1Gig connection, but I think better over powered than under powered and I will be keeping it for several years. It runs cool and silent and at idle draws around 5 watts. Maxes out my 1 Gig connection no problem with room to spare.
Standard User Pheasant
(experienced) Tue 27-Apr-21 10:11:12
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: E300] [link to this post]
 
Hehe sounds like you’ve horsepower in that box to run it as a PPPoE access concentrator! 😏
Standard User candlerb
(fountain of knowledge) Tue 27-Apr-21 10:18:54
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
Are you running the latest pfSense CE?

Does your J1900 box have dual integrated NICs for WAN/LAN, or are you using a USB-attached NIC or similar, or are you using VLANs?

I have some N2820-based NUCs (DN2820FYKH) running Linux, similar to your J1900. They can easily fill a gigabit of regular ethernet on the built-in NIC but I haven't tried them with PPPoE.

If you just want basic PPPoE routing and not to spend any money, then I'd certainly suggest giving OpenWrt a go. The Linux kernel may do a better job of PPPoE than the FreeBSD one.

If you like pfSense and want to continue using it, then get some more powerful hardware. You could even consider the official Netgate appliances, although they're not cheap.

If you want to buy a dedicated router appliance, the Mikrotik RB4011 has four ARM cores and has no problem filling a gigabit (it has 10 x 1G ports and one 10-gig port). Just note that the built-in switching hardware isn't VLAN-aware, so if you do clever stuff like trunking VLANs it will be switching in software. Even then, it can switch nearly a gigabit on a single core anyway.
Standard User jimbof
(regular) Tue 27-Apr-21 10:24:13
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
If you turn off intrusion / threat detection and smart queues / QOS then the Ubiquiti Unifi USG-4Pro is capable of routing my 900/115 Zen connection. With 900/115 the QOS seems for the large part redundant, as in real use I rarely see saturation.

It's also able to do site-to-site VPN to a USG-3 at my office at between 7 and 9MB / sec (slower in one direction, which I guess is the fault of the less powerful USG-3)

Perhaps there is an Edgerouter product from Ubiquiti that could satisfy your requirement?

As a workaround, if you think the issue is PPPoE, could you try and use some other small device with hardware PPPOE offload as a PPPoE to IP bridge... maybe even the Zen supplied Fritz could be bent to your will...
Standard User danielhyde
(member) Tue 27-Apr-21 14:22:04
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Pheasant] [link to this post]
 
Mine sounds even more overkill, I'm using an old Xeon E3-1240V3 that is about 25W at idle

Thanks
Dan
Standard User dogcat
(learned) Tue 27-Apr-21 14:44:14
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
J4105 / J5005 based boards can handle 1gbps PPPoE. They are the next generation on from the j1900 and offer double the single threaded speed, along with more cores, etc and use near enough no power. There are a few USFF /Thin client machines that have them made by Dell, HP, Fujitsu etc that also have an expansion slot and can be found second hand - problem is many people online have caught wind of this so they usually go for £200+ which is a lot for a second hand machine with their specs.

If it doesn't have to be very small just get a SFF build Dell or Fujitsu, with your budget dictating how recent a generation. Best value if you really don't care about space are Xeon V1/V2 based small towers, they can be had for around £100 e.g. Dell PowerEdge T110 II - complete overkill though, and will use 5x or more power of a J4015/J5005 based system.

Edited by dogcat (Tue 27-Apr-21 14:45:09)

Standard User mikehiow
(committed) Tue 27-Apr-21 15:24:50
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
Thanks for all the advice.

I've been playing with all sorts of ideas. The best performer so far? A RPI4 with OpenWRT which can push full 900/100.

I just don't trust a USB ethernet adapter or a little Pi running from MicroSD, however.

I've ordered a Mikrotik RB4011iGS+RM in the hope that it should be ultra-reliable once setup and still less money than the APU2 that's being returned.

Goodbye Hyperoptic. Now VM200 frown
Standard User jimbof
(regular) Tue 27-Apr-21 19:26:57
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
Pi compute module 4 supports onboard EMMC which is considered perhaps a little hardier than many SD cards, and also has a 1x PCIexpress interface on the docking header. There might be gigabit carriers over PCI express at some point.

Still might struggle though with PPPoE and getting the gigabit in and out at the same time.
Standard User Spud2003
(fountain of knowledge) Wed 28-Apr-21 04:11:15
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
The Mikrotik will also shift 2Gb/s IPsec VPN throughput, I love mine(also free OS updates unlike Cisco etc).
Standard User Spud2003
(fountain of knowledge) Wed 28-Apr-21 04:17:57
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: danielhyde] [link to this post]
 
My personal preference is a CPU that can be passively cooled - I'm using a Ryzen GE series CPU(Pro 3200GE) that has a 35W TDP and works with an Alpine heatsink. Fans always fail from my experience.
Standard User jimbof
(regular) Wed 28-Apr-21 08:03:48
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
Does look like an amazing little beasty.
Standard User Pheasant
(experienced) Wed 28-Apr-21 08:31:03
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
In reply to a post by Spud2003:
The Mikrotik will also shift 2Gb/s IPsec VPN throughput, I love mine(also free OS updates unlike Cisco etc).

We should start a separate thread on that very subject. Obviously headline throughput is quite dependent on IPsec configuration. I've run some bandwidth tests on Miktriotik CCR to CCR site to site VPN's and the results are interesting.
Standard User danielhyde
(member) Wed 28-Apr-21 11:01:14
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
In reply to a post by Spud2003:
My personal preference is a CPU that can be passively cooled - I'm using a Ryzen GE series CPU(Pro 3200GE) that has a 35W TDP and works with an Alpine heatsink. Fans always fail from my experience.


One thing i'd say on this is as long as you get a decent motherboard you can limit the TDP of the CPU to 35W so you wouldn't necessarily need the E version and could use any Ryzen CPU/APU

Thanks Dan
Standard User mikehiow
(committed) Wed 28-Apr-21 12:22:55
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
Well it certainly does the job for my connection.

Or at least it does now I know never to remove ether2 from bridge!

https://www.thinkbroadband.com/speedtest/16196083178...

Goodbye Hyperoptic. Now VM200 frown
Standard User Spud2003
(fountain of knowledge) Wed 28-Apr-21 21:43:22
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Pheasant] [link to this post]
 
The somewhat exotic multi-core Mikrotik CCR routers have multiple encryption modules shared between multiple cores AFAIR. So I think you may still get fast, but limited, VPN performance when a shared CCR encryption module is maxed out. The RB4011 may well outperform a nominally more powerful CCR(in VPN use) when both use a single encryption module.
Standard User Spud2003
(fountain of knowledge) Wed 28-Apr-21 21:50:26
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
There are multiple ways to administer it, I use Winbox for what it's worth(don't forget to backup your working configs). smile
Standard User chriscdotcodotuk
(regular) Wed 28-Apr-21 21:52:40
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
I wondered if my Edgerouter Lite would be able to cope with my connection so set it up today to give it a blast and sure enough the little guy is quite happy with the BT Fiber 900/100 connection acting as a PPPoE controller.

I even load balanced it with a 4G connection from Three so I could see a 986mb download speed and a healthy 178mb upload speed.

The CPU wasn't maxed out either which I was quite impressed with but the single gigabit link to the main switch was pretty much leaking 1's and 0's onto the attic floor. Total saturation!

I turned off the 4G connection as it was pointless and I don't have QOS not because it's not something I require.

Personally I use a VPN on one machine and as it is software based it is quite happy at 5-600mb down and happy to use most of the upload but I don't have a need to use a VPN on the actual wider network.

All in all I'm happy I've got to remove the BT super hub from my network. Just a shame the ONT's dont have native POE otherwise it'd be POEPPPoE (I've got injectors so technically I do have that)

https://www.speedtest.net/result/d/dd302aff-dae4-4e9...

BT Fiber - 920 down, 100 up.
Standard User ft247
(regular) Wed 28-Apr-21 22:15:23
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Pheasant] [link to this post]
 
In reply to a post by Pheasant:
I've run some bandwidth tests on Miktriotik CCR to CCR site to site VPN's and the results are interesting.


I'd be interested in this if you get a chance to share. So far in single tunnel L2TP/IPsec I'm hitting limits around 230Mbit with a CCR1009 to CHR. 1100AHx4 to CHR seems to do a bit better, up to 300Mbit and after that my connection can't do any better.
Standard User candlerb
(fountain of knowledge) Thu 29-Apr-21 08:05:13
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: ft247] [link to this post]
 
In a datacentre application, I have more powerful Mikrotik boxes - 36-core CCR1036-8G-2SplusEM with gigabit uplinks (one uplink is 10G). However when I tried them for site-to-site GRE-over-IPSEC I didn't get more than about 400Mbps. This was for a single iperf3 stream over a single tunnel, both data centres in London. As far as I could see all the traffic hit a single core. As this was a live network I wasn't able to play too much. Turning on fastpath didn't help.

In the end I deployed separate Linux VMs running Wireguard, and now I can easily fill a gigabit of site-to-site VPN traffic. In fact, running a pair of Wireguard test VMs connected back-to-back on the same host, with only 2 vCPUs each, I can get 2.5Gbps+ of iperf3 through them.

Wireguard has been backported to the mainline Ubuntu 18.04/20.04 kernels, so it's now easy to deploy without building your own module. There was also an implementation made by Netgate for pfSense, but it was kicked out of FreeBSD over code quality issues. Hopefully a better version will make it there soon.

The other problem I have with Mikrotik IPSEC is that they don't have VTI mode, which I use everywhere as it allows BGP failover between tunnels and real links. That's why I ended up having to try GRE-over-IPSEC.

I'd say Mikrotik is probably fine for a VPN concentrator, where you have lots of separate IPSEC tunnels coming in - in which case, the tunnels will be spread over the cores.
Standard User mikehiow
(committed) Thu 29-Apr-21 10:07:34
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
Are you using yours with PPPoE?

I seem to be getting a fair amount of packet loss when saturating the connection.

I can create a simple queue to limit max speed, but I've got to knock around 10% off real world speeds to eliminate the packet loss.

Goodbye Hyperoptic. Now VM200 frown
Standard User ft247
(regular) Thu 29-Apr-21 13:52:59
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: candlerb] [link to this post]
 
In reply to a post by candlerb:
I'd say Mikrotik is probably fine for a VPN concentrator, where you have lots of separate IPSEC tunnels coming in - in which case, the tunnels will be spread over the cores.


That's my feeling too. With the new CCR2004 being ARM based, I wonder if Tilera (CCR10xx) will remain a development priority for them.
Standard User Spud2003
(fountain of knowledge) Thu 29-Apr-21 15:36:27
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
I'm on VDSL not FTTP so I am not running at your throughput, I'm just using the official tech specs. Try the Mikrotik user forums, they should sort out any issues - https://forum.mikrotik.com/index.php
Standard User aidanh
(newbie) Thu 29-Apr-21 16:22:34
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
This is normal behaviour when you saturate a link. The only way to properly control it is to make sure you are the bottleneck (ideally with something smarter than a dumb queue).

Standard User candlerb
(fountain of knowledge) Thu 29-Apr-21 16:30:04
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Spud2003] [link to this post]
 
I have used both RB4011 and the smaller hEX PoE with PPPoE, no problems. I enable "baby jumbos" so the MTU on the PPPoE interface is 1500, and on the physical link is 1508.

The hEX PoE has a single MIPS core. When I used it on my 300/30 FTTPoD, with fasttrack, routing 300Mbps of iperf3 traffic used about 35% CPU. Without fasttrack, e.g. when doing IPv6, it would saturate the CPU and not quite reach 300Mbps.

I still keep it as a backup, and as a PoE switch for powering my APs. It can do VLAN switching in hardware, which makes it actually faster at switching than the RB4011.
Standard User Pheasant
(experienced) Fri 30-Apr-21 07:27:03
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: candlerb] [link to this post]
 
I ran a path MTU check over the PPPoE link by using the don’t fragment ping check trial and error method and it came to 1452. The actual MTU setting for the link is 1480. Running over IPSec the MTU is 1398

Edited by Pheasant (Fri 30-Apr-21 07:33:52)

Standard User Pheasant
(experienced) Fri 30-Apr-21 07:29:50
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: ft247] [link to this post]
 
In reply to a post by ft247:
In reply to a post by candlerb:
I'd say Mikrotik is probably fine for a VPN concentrator, where you have lots of separate IPSEC tunnels coming in - in which case, the tunnels will be spread over the cores.


That's my feeling too. With the new CCR2004 being ARM based, I wonder if Tilera (CCR10xx) will remain a development priority for them.

It’s looking that way. The single tunnel IPSec performance for the 2004 is about three times that of the Tilera boxes around 3Gbps. Aggregate throughout is a lot higher for the Tileras because of the sheer core count.
Standard User candlerb
(fountain of knowledge) Fri 30-Apr-21 07:34:47
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Pheasant] [link to this post]
 
In reply to a post by Pheasant:
I ran a path MTU check over the PPPoE link by using the don’t fragment ping check trial and error method and it came to 1452. The actual MTU setting for the link is 1480.


That's strangely low. Without tweaks, I would expect MTU 1492 for a PPPoE interface (1500 - 8 byte PPPoE header). You're not running a VPN or anything like that over it?

Also, how are you testing? Using Linux:
ping -Mdo -s1472 x.x.x.x # => MTU 1500 (20 bytes IP header + 8 bytes ICMP header + payload)
Standard User Pheasant
(experienced) Fri 30-Apr-21 14:23:13
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: candlerb] [link to this post]
 
In reply to a post by candlerb:
In reply to a post by Pheasant:
I ran a path MTU check over the PPPoE link by using the don’t fragment ping check trial and error method and it came to 1452. The actual MTU setting for the link is 1480.


That's strangely low. Without tweaks, I would expect MTU 1492 for a PPPoE interface (1500 - 8 byte PPPoE header). You're not running a VPN or anything like that over it?

Also, how are you testing? Using Linux:
ping -Mdo -s1472 x.x.x.x # => MTU 1500 (20 bytes IP header + 8 bytes ICMP header + payload)

Apologies! Upon investigation PPPoE link MTU was weirdly set as 1480 on the router i/f and should have been set as 1492 for the PPPoE link. With that said ping testing to 8.8.8.8 with a 1464 MSS/payload gives the expected result of 1492 on the PPPoE uplink.

Via IPsec the MSS/payload is 1398 or an MTU of 1426.
Standard User candlerb
(fountain of knowledge) Fri 30-Apr-21 18:35:49
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: Pheasant] [link to this post]
 
To do baby jumbos on Mikrotik:

Text
1
23
45
/interface ethernet
set <N> l2mtu=1526 mtu=1508 
/interface pppoe-clientset <N> max-mru=1500 max-mtu=1500


(P.S. To be strictly accurate, MSS refers to TCP segment size, not ICMP payload. IPv4+TCP header combined are usually 40 or 48 bytes, depending on whether the TCP timestamp option is present; so if you have IP MTU 1500 then the TCP MSS is 1460 or 1452. It's lower on IPv6 because of the larger IP headers)
Standard User nofappingway
(newbie) Tue 04-May-21 19:33:41
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: mikehiow] [link to this post]
 
Personally I use a Draytek 2850ac that's connected to a symmetrical gigabit FTTH connection from Hey! Broadband and a 75Mb vDSL from Plusnet simultaneously.

It runs both at full speed without issue in parallel.
Standard User smouty
(member) Wed 05-May-21 12:22:18
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: nofappingway] [link to this post]
 
Impressive for 10 year old hardware wink

Still using one of these at work but only for dsl.

OPNSense
PiHole
Unifi for Wifi
Standard User nofappingway
(newbie) Wed 05-May-21 12:32:35
Print Post

Re: Router hardware capable of routing 900mbps over PPPoE (F


[re: smouty] [link to this post]
 
Doh! Typo. Not a 2850.....and 2865 smile

Edited by nofappingway (Wed 05-May-21 12:33:48)

Pages in this thread: 1 | 2 | 3 | 4 | (show all)   Print Thread

Jump to