Future upgrade consideration advice wanted :: Home Networking, Internet Connection Sharing, etc.

Technical Discussion
>> Home Networking, Internet Connection Sharing, etc.

Pages in this thread: 1 | [2] | 3 | (show all)

Print Thread

Ixel
(experienced) Thu 26-Aug-21 19:45:35

Re: Future upgrade consideration advice wanted

[re: Pheasant] [link to this post]

Thanks. It was easier than I imagined it would be.

I've added two SFQ queues, one for PPPoE which caps the upstream at 100Mbit~ and another for now which caps the GRE tunnel to 400Mbit. Any higher than 400Mbit starts to show signs of an increasing ping (up to +100ms~ the original ping).

I was going to use fq_codel or perhaps even cake, but I've read about some stability issues when using those so I'm avoiding them for now. SFQ seems to do the job just fine.

That's still nice throughput, a lot better than I could ever get on the EdgeRouter Pro 8. If I can figure out where the bottleneck is and get more throughput then that's a bonus, otherwise I'm happy with that.

EDIT: Apparently the keepalive issue I had is a bug until v7.1rc1 - which fixes it.

Edited by Ixel (Thu 26-Aug-21 20:45:00)

Pheasant
(fountain of knowledge) Thu 26-Aug-21 22:50:51

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

Cool. How is CPU util. % on the box during testing?

I'm still runny 6.48 on CCR1's but have a couple of CCR2004's to setup on 7.

Ixel
(experienced) Thu 26-Aug-21 23:52:19

Re: Future upgrade consideration advice wanted

[re: Pheasant] [link to this post]

I see.

If fastpath, about 8% or so at 500Mbit~ GRE throughput. Otherwise if it's not fastpath, between 20% and 30% at the same GRE throughput.

Pheasant
(fountain of knowledge) Fri 27-Aug-21 00:44:01

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

Barely breaking a sweat using fastpath 😎

Ixel
(experienced) Fri 27-Aug-21 09:43:23

Re: Future upgrade consideration advice wanted

[re: Pheasant] [link to this post]

Indeed, and the OVH server has very low CPU usage too (which I'd expect). So the throughput bottleneck can't be CPU related.

I did a quick speed test on the OVH server using speedtest-cli:

Text
1 23 45 6	Selecting best server based on ping... Hosted by toob Ltd (London) [343.04 km]: 4.93 msTesting download speed Download: 1979.20 Mbit/sTesting upload speed Upload: 995.78 Mbit/s

It seems as if I can get more, as I imagined so, just need to figure out why I can't really go much beyond 400Mbit before the latency starts to rise a fair bit (before topping out around 570Mbit on a speed test). I'm wondering if I should perhaps see if a different type of tunnel makes any difference, e.g. IPIP (if that works with how I've configured GRE routing wise), but I have a hunch it may not make any difference assuming it worked.

EDIT: This is my latest config.

Text

# aug/27/2021 11:01:50 by RouterOS 7.1rc1
# software id = [redacted]#
# model = CCR2004-16G-2S+# serial number = [redacted]
/interface lteset [ find ] disabled=yes name=lte1
/interface ethernetset [ find default-name=ether9 ] name=ether1
set [ find default-name=ether10 ] name=ether2set [ find default-name=ether11 ] name=ether3
set [ find default-name=ether12 ] name=ether4set [ find default-name=ether13 ] name=ether5
set [ find default-name=ether14 ] name=ether6set [ find default-name=ether15 ] name=ether7
set [ find default-name=ether16 ] name=ether8set [ find default-name=ether1 ] name=ether9
set [ find default-name=ether2 ] name=ether10set [ find default-name=ether3 ] name=ether11
set [ find default-name=ether4 ] name=ether12set [ find default-name=ether5 ] name=ether13
set [ find default-name=ether6 ] name=ether14set [ find default-name=ether7 ] name=ether15
set [ find default-name=ether8 ] name=ether16/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 keepalive-timeout=60 max-mru=1492 max-mtu=1492 name="PPPoE Cerberus" user=x/interface gre
add !keepalive local-address=83.x.x.169 mtu=1468 name="OVH GRE Tunnel" remote-address=145.x.x.191/interface list
add name=LAN/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik/ppp profile
add change-tcp-mss=yes name=Cerberus only-one=yes use-mpls=no/queue type
add kind=sfq name=sfq/queue simple
add bucket-size=0.001/0.001 comment="RX/TX are reversed" max-limit=0/100M name="PPPoE Cerberus Queue" queue=sfq/sfq target="PPPoE Cerberus"add bucket-size=0.001/0.001 comment="RX/TX may also be reversed" max-limit=400M/400M name="OVH GRE Tunnel Queue" queue=sfq/sfq target="OVH GRE Tunnel"
/routing tableadd disabled=no name=666
/ip settingsset allow-fast-path=no
/ip addressadd address=192.168.1.1/24 comment=defconf interface=ether2 network=192.168.1.0
add address=198.x.x.1/24 comment="OVH GRE Tunnel" interface=ether3 network=198.x.x.0/ip dns
set servers=1.1.1.1,1.0.0.1/ip firewall address-list
add address=192.168.1.0/24 list=LANadd address=198.x.x.0/24 list=OVH
/ip firewall filteradd action=accept chain=forward comment="Accept established and related" connection-state=established,related
add action=drop chain=forward comment="Drop invalid" connection-state=invalidadd action=fasttrack-connection chain=forward hw-offload=no
add action=accept chain=input comment="Accept established and related" connection-state=established,relatedadd action=drop chain=input comment="Drop invalid" connection-state=invalid
add action=accept chain=input comment="Accept ICMP" dst-limit=5,10,dst-address/1m40s in-interface=all-ppp limit=5,10:packet protocol=icmpadd action=accept chain=input comment="Accept GRE traffic from 145.x.x.191" in-interface=all-ppp protocol=gre src-address=145.x.x.191
add action=drop chain=input comment="Drop all traffic from PPP" in-interface=all-pppadd action=drop chain=input comment="Drop TCP to 198.x.x.1 from WAN" dst-address=198.x.x.1 in-interface="OVH GRE Tunnel" protocol=tcp
add action=drop chain=input comment="Drop UDP to 198.x.x.1 from WAN" dst-address=198.x.x.1 in-interface="OVH GRE Tunnel" protocol=udpadd action=accept chain=input comment="Accept everything else"
/ip firewall natadd action=masquerade chain=srcnat src-address=192.168.1.0/24
/ip firewall service-portset ftp disabled=yes
set tftp disabled=yesset irc disabled=yes
set sip disabled=yes/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway="OVH GRE Tunnel" pref-src="" routing-table=666 scope=30 suppress-hw-offload=no target-scope=10add disabled=no dst-address=198.x.x.0/24 gateway=ether3 routing-table=666 suppress-hw-offload=no
add disabled=no dst-address=192.168.1.0/24 gateway=ether2 routing-table=666 suppress-hw-offload=no/ip service
set telnet disabled=yesset ftp disabled=yes
set www disabled=yesset api disabled=yes
set api-ssl disabled=yes/routing rule
add action=lookup disabled=no src-address=198.x.x.0/24/system clock
set time-zone-name=Europe/London/system ntp client
set enabled=yes/system ntp client servers
add address=80.86.38.193add address=143.210.16.201
add address=178.79.160.57add address=217.114.59.3
add address=87.117.251.3add address=109.237.17.140
add address=178.79.162.34add address=188.39.98.165

I've upgraded to v7.1rc1 to see if it would make any difference, I didn't imagine it would and sadly I was right.

EDIT 2: Tried IPIP tunnel, slightly worse (520Mbit~). Back on GRE tunnel again. I have a feeling I may have to ask the MikroTik forum about this as I'm a little stumped. I'm happy with the speed if this is the most I can get (about 400Mbit before ping starts to significantly rise), just I'd like to know what the cause of the supposed bottleneck is. I guess I hate having an unsolved mystery tongue

, which this is.

Edited by Ixel (Fri 27-Aug-21 18:11:54)

Ixel
(experienced) Sat 28-Aug-21 00:36:49

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

It looks like this could currently be a bug in RouterOS 7.x.

Went to an unofficial Discord to discuss this and someone there asked me to do a CPU profiling, and that revealed something I overlooked. On the EdgeRouter Pro 8 it has 2 CPU cores and the GRE tunnel was able to use 100% CPU if I didn't rate limit the throughput. However, on the MikroTik the profiler reveals that the GRE tunnel (during a speed test) is only using one CPU core and it's not fully using that CPU core either. They recommended I report this as a bug, which I've now done, although it might be a long time before it's actually fixed I've been warned. Will see what support come back with.

Pheasant
(fountain of knowledge) Sat 28-Aug-21 08:51:10

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

Interesting. The base throughput is still very decent even though it’s being held back from using more CPU. I’m not sure I’d want it being able to completely saturate CPU though. Still a lot of work to do on 7 to get it to the same level of maturity of 6. It’s a full bottom up rewrite though, desperately needed as 6 is now creaking and hamstrung on an outdated Linux core.

Ixel
(experienced) Sat 28-Aug-21 10:06:53

Re: Future upgrade consideration advice wanted

[re: Pheasant] [link to this post]

Indeed, I'm very impressed with what a single core can achieve. If and when they fix this issue one day, the extra throughput I consider to be a bonus. I'm satisfied with the current throughput and at least know what the likely cause of the bottleneck is at the moment. If they do fix this then I doubt I could push the GRE tunnel enough to fully utilise the CPU anyway, as I presume it would become capable of somewhere around 1.5Gbit+. In the unlikely event it was capable of doing so though... I'd just rate limit the throughput.

It's certainly been an interesting experience, I didn't imagine I'd adapt to RouterOS's interface and configuration as soon as I did.

Edited by Ixel (Sat 28-Aug-21 10:08:03)

Pheasant
(fountain of knowledge) Sat 28-Aug-21 10:42:47

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

I was fairly confident about the expected GRE capability, given that it’s got enough grunt to do IPSec at > 3 Gbps over a single tunnel or up to 256 tunnels. It’s a shame there’s a limitation at the moment in RouterOS 7 with GRE performance. I’d expect if it was addressed there’s no reason performance couldn’t meet or exceed that of IPSec.

prlzx
(experienced) Sat 28-Aug-21 11:21:28

Re: Future upgrade consideration advice wanted

[re: Ixel] [link to this post]

I'm late to the thread, but when you were using the EdgeRouter Pro, did you enable all the relevant hardware offload features as they aren't on by default (and GRE has its own setting within that)?

I only ask as normally an EdgeRouter is only CPU-bound if not offloading traffic, and it's quite common for people to enable all manner of QoS features without realising some features aren't eligible for offload because they require processing on the CPU.

For anyone who runs an EdgeRouter who isn't aware I'll post the link as it's an important factor for model selection including if you plan to use PPPoE, VLANs, IPSec, GRE, or bridging ports (instead of using a model with switched ports).

https://help.ui.com/hc/en-us/articles/115006567467-E...

That said if you were on the Ubiquiti forums I'd be surprised if it wasn't the first question people asked - it's such a recurring thing for people to ask why they have high CPU usage and don't reach wire speed.

In an extreme example an audio-visual installer had sold someone a system including an edgerouter 8-port but were using it to bridge all the ports, and the company who took over the contract didn't want to tell the customer why their "expensive" router performed less well than a basic SOHO router (with an actual switch built-in), nor replace it with a £20 switch and admit they were mis-sold, instead demanding that the forum provide a way to make it go faster.

prlzx on Zen: FTTC (VDSL) at ~40Mbps / 10Mbps
with IP4/6 (no v6? - not true Internet)

Edited by prlzx (Sat 28-Aug-21 12:17:35)

Pages in this thread: 1 | [2] | 3 | (show all)

Print Thread

Jump to