|
|
|
Hi,
Recently I purchased a Unifi Dream Machine Pro, sadly it was a disappointment though. Firmware is nowhere near finished imo, well, it's a somewhat shared opinion over at the Ubiquiti forum I can see. Some basics that I took for granted, like NAT settings, don't even exist. I've sent it back to the seller pending a refund and am now back on my EdgeRouter Pro 8 again (which I was going to sell).
Now, slight possible rant aside, I'm looking for some advice. I've been used to VyOS (or EdgeOS) for some time but if need be I will learn a new interface and operating system. I'm looking for a router with at least four RJ45 ports but has sufficient processing power to handle gigabit PPPoE and hopefully at least 500Mbit GRE throughput. The EdgeRouter Pro 8 can handle about 100Mbit GRE throughput at 50%~ CPU usage, so I've currently capped the tunnel to 100/100. Not shabby, but if it's possible to get a bit more out of it then I would like to.
I have considered running VyOS, or perhaps pfSense or opnSense on my Unraid system as a virtual machine, but sadly it only has two ethernet ports (both 10Gbit capable but running at 1Gbit at the moment). I thought about VLANs as my EdgeSwitch could handle that, perhaps isolating the PPPoE connection to its own VLAN or something? I've never played with VLANs though, a new area for me. Other option I suppose is purchasing a four port PCI-E card, still considerably cheaper than the cost of the UDM Pro.
Any advice please?
One other thing I'm considering is whether it's perhaps a good idea to buy a network/firewall appliance (also known as a mini PC with a few RJ45 ports on). I could install VyOS on that, for example. I just need to make sure the CPU is more powerful than the EdgeRouter Pro 8's one and that hopefully it meets my expectations on PPPoE and GRE throughput.
Thanks.
|
|
|
I have a symmetrical gigabit FTTP connection and I use a DrayTek 2865 with it. I get line speed using PPPoE including filtering using its Firewall (IP filter). It has 5 Gbe ports and a lot features.
https://www.draytek.co.uk/products/business/vigor-28...
|
|
|
Thanks for replying.
That looks impressive. It's been a long time since I've used DrayTek, they generally make good routers though. I would need to double check, perhaps by contacting DrayTek, that it supports GRE without IPsec (as I don't need IPsec) and presumably a GRE tunnel can be setup without assigning it an IP address (all I do currently is just tell it the local IP address and remote/peer IP address, then route a specific IP subnet through it using some policy routing. Presumably this device could do that without much hassle.
This is what I currently do on the EdgeRouter:
| Text | 1
23
45
6 | /sbin/ip tunnel add gre1 mode gre remote 51.x.x.62 local 83.x.x.169 ttl 255
/sbin/ip link set gre1 up/sbin/ip rule add from 198.x.x.0/24 table 666
/sbin/ip route add default dev gre1 table 666/sbin/ip route add 198.x.x.0/24 dev eth3 table 666
/sbin/ip route add 192.168.1.0/24 dev eth1 table 666 |
eth3 then has an IP address of 198.x.x.1/24 which acts as a gateway IP address. eth1 is just my LAN devices so they have a direct route locally.
Definitely one I will further look into and contact DrayTek about before potentially buying it. I have a 2862 in the cupboard at the moment, which I used to use at one point when I had VDSL2. Fairly reliable.
Edited by Ixel (Tue 24-Aug-21 12:23:52)
|
|
Register (or login) on our website and you will not see this ad.
|
|
|
|
As far as I'm aware DrayTek routers do not support that.
Thanks
Dan
|
|
|
Looking at some of DrayTek's knowledge base I can at least see it looks like IPsec is optional for the GRE tunnel. How easy it is to setup the necessary policy routing and whether I can setup the GRE tunnel without assigning it an IP address is another question.
From what I understand on DrayTek's website, the 2862 may support GRE. I will get mine out of the cupboard, install the latest firmware and see if I can get any further insight into what's possible.
If it's not possible to do this on the DrayTek router then I guess my only other option is to ultimately buy some kind of mini PC with multiple ethernet ports on, then install something like VyOS. Could be more costly but I guess my use case isn't a common one.
EDIT: Looks like it may also be possible to setup the routing similar or perhaps even identical to how I have it at the moment. I will try to see if I can test this out on my 2862 before I buy a 2865.
Edited by Ixel (Tue 24-Aug-21 17:55:21)
|
|
|
One of the multi-core ARM based Mikrotik routers should be able to get you decent GRE performance on a gig WAN link. Depends on your budget but they released two new routers recently...
RB5009UG+S+IN which comes in at £126+VAT from Eurodk
Alternatively if the name of the game is pure throughput, whether IPsec or straight, there is now an "all copper" CCR2004-16G-2S+ beastie which will give you multi-gig GRE performance, for around £268+VAT from Eurodk
|
|
|
The £268+VAT is within my budget, the UDM Pro cost me a bit more so that's fine. That looks like a beast, certainly an impressive piece of kit and I love the fact it's rackmountable too. I've never used RouterOS so it's something I will have to read up on and if there's an online demo then I will also play around with that. Else failing that I'm sure there's plenty of YouTube videos talking about it.
Wifi Stock UK apparently has them in stock for Friday delivery at the moment (probably until some point tomorrow), so where possible I'll order it from a UK store as I'm not sure if I'll get lumbered with customs delays or perhaps an extra charge if I buy from an EU store for that amount of money.
EDIT: Looking at the demo it doesn't seem all that complicated to learn. Things appear to be clearly labeled. Certainly very customisable from what I can see.
EDIT 2: Well I'm almost certain that's what I'll now be going for. In the unlikely event I get stuck with setting up what I want to do then I see they have a quite an active forum which is also nice. Thanks very much for the suggestion, it looks like the perfect solution for my needs!
Edited by Ixel (Tue 24-Aug-21 21:07:12)
|
|
|
One warning about those new Mikrotik Routers, they are using version 7 of their OS, which is still in Beta, although a release candidate has finally been posted. Why they are releasing them before the OS is ready is questionable.
They are both amazing routers though, just be aware the first few weeks could be buggy until the OS is ready.
Edited by deleted (Thu 26-Aug-21 12:55:55)
|
|
|
Sounds a little like the development of the UDM Pro's OS, but not anywhere near as bad as the UDM Pro's OS.
All being well the 'stable' version that came installed will be stable enough for the moment. I'll avoid the beta and release candidate if possible. I agree though, it's a little silly releasing something that's not quite ready.
---
Also to post an update to say that I received the item shortly before 12pm, the UPS man seemed to pretty much be Roadrunner though haha. I've never seen someone get out of a delivery van and then back in the van so quick.
It looks impressive in the cabinet, albeit everything else is in the cabinet is painted black so doesn't match.
I believe I've set it up correctly, at least to get things started anyway, I won't know until later this afternoon when I connect the LAN cables to the appropriate ports. I like how it didn't cause disruption by plugging it in to my switch, e.g. no IP address conflict, connected with Winbox via the MAC address.
Below is my current configuration which I hope will work without any hassle (sensitive information redacted of course):
| Text | 1
23
45
67
89
1011
1213
1415
1617
1819
2021
2223
2425
2627
2829
3031
3233
3435
3637
3839
4041
4243
4445
4647
48 | /interface pppoe-client
add add-default-route=yes interface=ether1 keepalive-timeout=60 name=\ "PPPoE Cerberus" password=x use-peer-dns=yes user=x
/interface greadd mtu=1468 name="OVH GRE Tunnel" remote-address=145.x.x.191
/routing tableadd disabled=no name=666
/ip addressadd address=192.168.1.1/24 comment=defconf interface=ether2 network=\
192.168.1.0add address=198.x.x.1/24 comment="OVH GRE Tunnel" interface=ether3 \
network=198.x.x.0/ip dns
set servers=1.1.1.1,1.0.0.1/ip firewall address-list
add address=192.168.1.0/24 list=LANadd address=198.x.x.0/24 list=OVH
/ip firewall filteradd action=accept chain=forward comment="Accept established and related" \
connection-state=established,relatedadd action=drop chain=forward comment="Drop invalid" connection-state=invalid
add action=fasttrack-connection chain=forward hw-offload=yesadd action=accept chain=input comment="Accept established and related" \
connection-state=established,relatedadd action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=accept chain=input comment="Accept ICMP" in-interface=all-ppp \ protocol=icmp
add action=accept chain=input comment=\ "Accept GRE traffic from 145.x.x.191" in-interface=all-ppp protocol=\
gre src-address=145.x.x.191add action=drop chain=input comment="Drop all other traffic via PPP" in-interface=\
all-pppadd action=accept chain=input
/ip firewall natadd action=masquerade chain=srcnat src-address=192.168.1.0/24
/ip firewall service-portset tftp disabled=yes
set irc disabled=yes/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway="OVH GRE Tunnel" \ pref-src="" routing-table=666 scope=30 suppress-hw-offload=no \
target-scope=10add disabled=no dst-address=198.x.x.0/24 gateway=ether3 routing-table=666 \
suppress-hw-offload=noadd disabled=no dst-address=192.168.1.0/24 gateway=ether2 routing-table=666 \
suppress-hw-offload=no/routing rule
add action=lookup disabled=no src-address=198.x.x.0/24 table=666 |
My routing setup is based on the following commands I originally used on the EdgeRouter:
| Text | 1
23
45
6 | /sbin/ip tunnel add gre1 mode gre remote 145.x.x.191 local 83.x.x.169 ttl 255
/sbin/ip link set gre1 up/sbin/ip rule add from 198.x.x.0/24 table 666
/sbin/ip route add default dev gre1 table 666/sbin/ip route add 198.x.x.0/24 dev eth3 table 666
/sbin/ip route add 192.168.1.0/24 dev eth1 table 666 |
ether1 is the PPPoE port
ether2 is the LAN port (192.168.1.1/24)
ether3 is the GRE tunnel port to OVH (198.x.x.1/24)
All being well I've implemented the policy routing correctly on this device. I still need to possibly sort out some QoS on the upstream and perhaps fine tune the firewall rules. It's hopefully a start though. I'll update again later. If anyone happens to notice any issues with my configuration in the meantime please comment! Thanks.
---
EDIT: All working it seems, went easier than I expected. Just had one setting incorrect which presumably disabled a route until it was corrected. Keepalive had to be disabled on the GRE tunnel. After that it worked. Speed test got me about 570Mbps down, so I need to do more fine tuning and perhaps check the OVH server to see if there's a bottleneck on that side. Either way it's far superior to the performance I was able to get out of the EdgeRouter Pro 8, so if 570Mbps is the best I can get, although I think that's unlikely by a long shot, I'm still pleased with the outcome.
Edited by Ixel (Thu 26-Aug-21 16:28:42)
|
|
|
|
Well done on getting the setup running so quickly. I haven't looked into your config I must admit, but that's pretty decent throughput off the bat.
|
|
|
Thanks. It was easier than I imagined it would be.
I've added two SFQ queues, one for PPPoE which caps the upstream at 100Mbit~ and another for now which caps the GRE tunnel to 400Mbit. Any higher than 400Mbit starts to show signs of an increasing ping (up to +100ms~ the original ping).
I was going to use fq_codel or perhaps even cake, but I've read about some stability issues when using those so I'm avoiding them for now. SFQ seems to do the job just fine.
That's still nice throughput, a lot better than I could ever get on the EdgeRouter Pro 8. If I can figure out where the bottleneck is and get more throughput then that's a bonus, otherwise I'm happy with that.
EDIT: Apparently the keepalive issue I had is a bug until v7.1rc1 - which fixes it.
Edited by Ixel (Thu 26-Aug-21 20:45:00)
|
|
|
|
Cool. How is CPU util. % on the box during testing?
I'm still runny 6.48 on CCR1's but have a couple of CCR2004's to setup on 7.
|
|
|
|
I see.
If fastpath, about 8% or so at 500Mbit~ GRE throughput. Otherwise if it's not fastpath, between 20% and 30% at the same GRE throughput.
|
|
|
|
Barely breaking a sweat using fastpath 😎
|
|
|
Indeed, and the OVH server has very low CPU usage too (which I'd expect). So the throughput bottleneck can't be CPU related.
I did a quick speed test on the OVH server using speedtest-cli:
| Text | 1
23
45
6 | Selecting best server based on ping...
Hosted by toob Ltd (London) [343.04 km]: 4.93 msTesting download speed
Download: 1979.20 Mbit/sTesting upload speed
Upload: 995.78 Mbit/s |
It seems as if I can get more, as I imagined so, just need to figure out why I can't really go much beyond 400Mbit before the latency starts to rise a fair bit (before topping out around 570Mbit on a speed test). I'm wondering if I should perhaps see if a different type of tunnel makes any difference, e.g. IPIP (if that works with how I've configured GRE routing wise), but I have a hunch it may not make any difference assuming it worked.
EDIT: This is my latest config.
| Text | 1
23
45
67
89
1011
1213
1415
1617
1819
2021
2223
2425
2627
2829
3031
3233
3435
3637
3839
4041
4243
4445
4647
4849
5051
5253
5455
5657
5859
6061
6263
6465
6667
6869
7071
7273
7475
7677
7879
8081
8283
8485
8687
8889
9091
9293
9495
| # aug/27/2021 11:01:50 by RouterOS 7.1rc1
# software id = [redacted]#
# model = CCR2004-16G-2S+# serial number = [redacted]
/interface lteset [ find ] disabled=yes name=lte1
/interface ethernetset [ find default-name=ether9 ] name=ether1
set [ find default-name=ether10 ] name=ether2set [ find default-name=ether11 ] name=ether3
set [ find default-name=ether12 ] name=ether4set [ find default-name=ether13 ] name=ether5
set [ find default-name=ether14 ] name=ether6set [ find default-name=ether15 ] name=ether7
set [ find default-name=ether16 ] name=ether8set [ find default-name=ether1 ] name=ether9
set [ find default-name=ether2 ] name=ether10set [ find default-name=ether3 ] name=ether11
set [ find default-name=ether4 ] name=ether12set [ find default-name=ether5 ] name=ether13
set [ find default-name=ether6 ] name=ether14set [ find default-name=ether7 ] name=ether15
set [ find default-name=ether8 ] name=ether16/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 keepalive-timeout=60 max-mru=1492 max-mtu=1492 name="PPPoE Cerberus" user=x/interface gre
add !keepalive local-address=83.x.x.169 mtu=1468 name="OVH GRE Tunnel" remote-address=145.x.x.191/interface list
add name=LAN/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik/ppp profile
add change-tcp-mss=yes name=Cerberus only-one=yes use-mpls=no/queue type
add kind=sfq name=sfq/queue simple
add bucket-size=0.001/0.001 comment="RX/TX are reversed" max-limit=0/100M name="PPPoE Cerberus Queue" queue=sfq/sfq target="PPPoE Cerberus"add bucket-size=0.001/0.001 comment="RX/TX may also be reversed" max-limit=400M/400M name="OVH GRE Tunnel Queue" queue=sfq/sfq target="OVH GRE Tunnel"
/routing tableadd disabled=no name=666
/ip settingsset allow-fast-path=no
/ip addressadd address=192.168.1.1/24 comment=defconf interface=ether2 network=192.168.1.0
add address=198.x.x.1/24 comment="OVH GRE Tunnel" interface=ether3 network=198.x.x.0/ip dns
set servers=1.1.1.1,1.0.0.1/ip firewall address-list
add address=192.168.1.0/24 list=LANadd address=198.x.x.0/24 list=OVH
/ip firewall filteradd action=accept chain=forward comment="Accept established and related" connection-state=established,related
add action=drop chain=forward comment="Drop invalid" connection-state=invalidadd action=fasttrack-connection chain=forward hw-offload=no
add action=accept chain=input comment="Accept established and related" connection-state=established,relatedadd action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=accept chain=input comment="Accept ICMP" dst-limit=5,10,dst-address/1m40s in-interface=all-ppp limit=5,10:packet protocol=icmpadd action=accept chain=input comment="Accept GRE traffic from 145.x.x.191" in-interface=all-ppp protocol=gre src-address=145.x.x.191
add action=drop chain=input comment="Drop all traffic from PPP" in-interface=all-pppadd action=drop chain=input comment="Drop TCP to 198.x.x.1 from WAN" dst-address=198.x.x.1 in-interface="OVH GRE Tunnel" protocol=tcp
add action=drop chain=input comment="Drop UDP to 198.x.x.1 from WAN" dst-address=198.x.x.1 in-interface="OVH GRE Tunnel" protocol=udpadd action=accept chain=input comment="Accept everything else"
/ip firewall natadd action=masquerade chain=srcnat src-address=192.168.1.0/24
/ip firewall service-portset ftp disabled=yes
set tftp disabled=yesset irc disabled=yes
set sip disabled=yes/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway="OVH GRE Tunnel" pref-src="" routing-table=666 scope=30 suppress-hw-offload=no target-scope=10add disabled=no dst-address=198.x.x.0/24 gateway=ether3 routing-table=666 suppress-hw-offload=no
add disabled=no dst-address=192.168.1.0/24 gateway=ether2 routing-table=666 suppress-hw-offload=no/ip service
set telnet disabled=yesset ftp disabled=yes
set www disabled=yesset api disabled=yes
set api-ssl disabled=yes/routing rule
add action=lookup disabled=no src-address=198.x.x.0/24/system clock
set time-zone-name=Europe/London/system ntp client
set enabled=yes/system ntp client servers
add address=80.86.38.193add address=143.210.16.201
add address=178.79.160.57add address=217.114.59.3
add address=87.117.251.3add address=109.237.17.140
add address=178.79.162.34add address=188.39.98.165 |
I've upgraded to v7.1rc1 to see if it would make any difference, I didn't imagine it would and sadly I was right.
EDIT 2: Tried IPIP tunnel, slightly worse (520Mbit~). Back on GRE tunnel again. I have a feeling I may have to ask the MikroTik forum about this as I'm a little stumped. I'm happy with the speed if this is the most I can get (about 400Mbit before ping starts to significantly rise), just I'd like to know what the cause of the supposed bottleneck is. I guess I hate having an unsolved mystery  , which this is.
Edited by Ixel (Fri 27-Aug-21 18:11:54)
|
|
|
|
It looks like this could currently be a bug in RouterOS 7.x.
Went to an unofficial Discord to discuss this and someone there asked me to do a CPU profiling, and that revealed something I overlooked. On the EdgeRouter Pro 8 it has 2 CPU cores and the GRE tunnel was able to use 100% CPU if I didn't rate limit the throughput. However, on the MikroTik the profiler reveals that the GRE tunnel (during a speed test) is only using one CPU core and it's not fully using that CPU core either. They recommended I report this as a bug, which I've now done, although it might be a long time before it's actually fixed I've been warned. Will see what support come back with.
|
|
|
|
Interesting. The base throughput is still very decent even though it’s being held back from using more CPU. I’m not sure I’d want it being able to completely saturate CPU though. Still a lot of work to do on 7 to get it to the same level of maturity of 6. It’s a full bottom up rewrite though, desperately needed as 6 is now creaking and hamstrung on an outdated Linux core.
|
|
|
Indeed, I'm very impressed with what a single core can achieve. If and when they fix this issue one day, the extra throughput I consider to be a bonus. I'm satisfied with the current throughput and at least know what the likely cause of the bottleneck is at the moment. If they do fix this then I doubt I could push the GRE tunnel enough to fully utilise the CPU anyway, as I presume it would become capable of somewhere around 1.5Gbit+. In the unlikely event it was capable of doing so though... I'd just rate limit the throughput.
It's certainly been an interesting experience, I didn't imagine I'd adapt to RouterOS's interface and configuration as soon as I did.
Edited by Ixel (Sat 28-Aug-21 10:08:03)
|
|
|
|
I was fairly confident about the expected GRE capability, given that it’s got enough grunt to do IPSec at > 3 Gbps over a single tunnel or up to 256 tunnels. It’s a shame there’s a limitation at the moment in RouterOS 7 with GRE performance. I’d expect if it was addressed there’s no reason performance couldn’t meet or exceed that of IPSec.
|
|
|
I'm late to the thread, but when you were using the EdgeRouter Pro, did you enable all the relevant hardware offload features as they aren't on by default (and GRE has its own setting within that)?
I only ask as normally an EdgeRouter is only CPU-bound if not offloading traffic, and it's quite common for people to enable all manner of QoS features without realising some features aren't eligible for offload because they require processing on the CPU.
For anyone who runs an EdgeRouter who isn't aware I'll post the link as it's an important factor for model selection including if you plan to use PPPoE, VLANs, IPSec, GRE, or bridging ports (instead of using a model with switched ports).
https://help.ui.com/hc/en-us/articles/115006567467-E...
That said if you were on the Ubiquiti forums I'd be surprised if it wasn't the first question people asked - it's such a recurring thing for people to ask why they have high CPU usage and don't reach wire speed.
In an extreme example an audio-visual installer had sold someone a system including an edgerouter 8-port but were using it to bridge all the ports, and the company who took over the contract didn't want to tell the customer why their "expensive" router performed less well than a basic SOHO router (with an actual switch built-in), nor replace it with a £20 switch and admit they were mis-sold, instead demanding that the forum provide a way to make it go faster.
prlzx on Zen: FTTC (VDSL) at ~40Mbps / 10Mbps
with IP4/6 (no v6? - not true Internet)
Edited by prlzx (Sat 28-Aug-21 12:17:35)
|
|
|
|
Unfortunately yes, the hardware offloading options were enabled. I later found out on a forum post somewhere that apparently GRE tunnels on that router aren't offloaded, only GRE packets which are forwarded via the router to somewhere else or something. Sadly that wasn't useful for me as I wanted the GRE tunnel on the router itself.
Thanks for the suggestion anyway though!
|
Pages in this thread: 
Print Thread
