20G NAT Throughput

Greets, folks.

So a set of requirements. On the WAN side it's 2 x publicly addressed links each symmetrical and give or take 10G. These need to have per-flow load balancing.

On the LAN side it's 2 x 10G LAG via a port channel, with a single LAN-side IP address.

The Mikrotik CCR2116-12G-4S+ will just about manage and I can happily configure load balancing on it. I'd welcome other thoughts from people more familiar with this kinda hardware than me. The Mikrotik is my go-to device though this one won't leave much change from a grand so I'm open to options.

EDIT: Modified subject line due to error. Requirement is 20G of throughput, 40G of total transfer: 20 in, 20 out.

Edited by XGS_Is_On (Sun 05-Jun-22 20:22:24)

Dual 10G load balanced access circuits!! That on separate providers/tails? Blimey that must be over a grand per month before VAT 😎

I suppose if the MT box struggled for throughput you could upgrade to their flagship 2216 which looks like an absolute stonking 25G/100G beast…but utter overkill for a multi 10G access device!? In any event I’m guessing if the client is running dual 10G circuits, an extra grand or two capex for the box won’t bust the friendship

On setup…definitely run using FastTrack. Memory footprint for the state table with large numbers of users?? You’d probably know more than most here in general routing setups. Can’t think of other specific MT tuning tips - still running RoS 6 on very much yesterdays ‘second rate’ Tilera spec boxes. I think there have been some changes / enhancements to how NAT is handled with RoS 7

Maybe a post up on the MikroTik forum?

Netgate will probably be able to provide you with a platform that can run TNSR and achieve those numbers

Personal not professional use so have to be a little mindful of cost.

Unfortunate there are no 4 x SFP28 switches, they come with loads of SFP+ or GbE ports that aren't required.

What about terminating into an Intel NiC with dual 10G SFP+ ports into a decent spec x86-64 box running pfSense or OPNsense? Enough grunt?

In reply to a post by Pheasant:

What about terminating into an Intel NiC with dual 10G SFP+ ports into a decent spec x86-64 box running pfSense or OPNsense? Enough grunt?

It's an option. Need another dual port card for it.

I have a plan involving a 25G spine with 10G leaves. FS are going to love my optics bill - BiDi.

Yup of course, completely forgot the LAN side.

FS lead times seemed to have pushed out on their optics. Not surprising given events. 10 km 25G BiDi are kind-of OK-ish at around £75 a pop. Mind be great if they were £8 like 1G BiDi now...

I was thinking the same but, pfSense boxes made by Netgate don't hit 20Gbps with their top spec box.
It has an 8 core Intel Xeon D-1541 and maxes out just below 19Gbps on L3 Forwarding and Firewall performance.

Thanks Dan

To be fair Intel CPU's have come along a fair bit since the Xeon D-1541 was announced in late 2015. Take for example a modern Core i9-12900K desktop processor vs server processor ...6 years of progress.

https://cputechie.com/comparisons/intel-core-i9-1290...

https://browser.geekbench.com/processor-benchmarks
Multi-Core scores:
Xeon D-1541= 4,564
Core i9-12900K = 17,278

Single Core scores:
Xeon D-1541= 579
Core i9-12900K = 1,990

Edited by Pheasant (Mon 06-Jun-22 22:45:52)

Forgot to mention: balancing like this means per connection classification then PBR, so some CPU work to do running through and processing mangle rules to allocate each flow to a link and give each flow a connection mark, with pre-routing using the earlier marks to produce a routing mark for use in static routes, alongside masquerade and IP filtering.

Lots of work for the CPUs to do outbound, thankfully not so much inbound, and inbound can take the fast track path.

LAG out the front end also consumes CPU.

I have, however, taken care of the super fast desktop PC. I'll move a CCR2004 closer and take advantage of that router's 25G ports to give a clean 25 to the PC.

It begins.

https://i.postimg.cc/v8zM1cfF/20220607-195321.jpg

Looks lonely. Needs a mate 🤣

The mate is the 25G port on a Mikrotik CCR2004.

As long as I'm careful I can coax the required throughput from it. Will be fitting behind the CCR2116 basically taking 2 x 10G ports and via the wonders of ECMP feeding the 2116.

2116 will live purely to handle per-connection classification and balancing. Won't even be the default gateway for the LAN to keep inter-LAN traffic away from it.

The 2004 having no filter rules to speak of or anything smart to do other than split packets evenly between two routed links means it can fastpath everything.

Will only need the 2116 as new hardware, along with a couple of optics and a DAC.

NIC #2 purchased. Dedicated fibre for 25G in place. 25G DAC ordered. Next 25GBase-BX optics and will be good to go besides the 2116, which will be purchased once the circuits are confirmed.

The 10G network is fine until the first circuit goes in when I'll bring the 25G star live to offload the two busiest machines from the 10G LAN.

Debating whether to borrow a fibre from the 10G to connect to the dedicated 25G or run the new cable the rest of the way. May well do the latter and at the same time install WiFi 6e. Keeps the resilient 10G ring in place and physically bypasses it via bridge interface on the 2004.

Will have a bridge interface covering the 2 25G ports on the router and the two 10G ports and use that bridge for VRRP. Right now it's a port channel but that will not be required and will take some load off the CPU.

Only really issue is that the 25G devices have to go through the 2004 to get to the rest of the LAN but that's unlikely to be an issue. The highest bandwidth end devices on the LAN with those offloaded are 2.5G. Would need to saturate the Internet connections and both 2.5G devices to hit the throughput limit on the 2004.

I could do it far cleaner but this'll work. Those of us that don't have unlimited budgets have to spend wisely.

Sounds good. Got a network drawing handy? I’m a simpleton and like pictures 😎

Yep, sent PM with a few of them.

In reply to a post by XGS_Is_On:

Those of us that don't have unlimited budgets have to spend wisely.

What will be total the monthly ongoing costs for this?

In reply to a post by Ripley:

What will be total the monthly ongoing costs for this?

Unpleasant.

But less than you might think. Most of the capacity is burst / shared so it won't come close to the costs of the equivalent dedicated access.

The Internet bill has historically been high anyway from use of multiple providers for redundancy, justified to at least some extent by how much of my work happens at home.

EDIT: The capacity is really, really shared. It's incredibly unlikely but combined throughput could, in theory, drop to sub-300 Mbit. That isn't going to happen, a number of things that would make Internet access quite irrelevant like huge natural disasters or nuclear war are way more likely, but it's not impossible.

Edited by XGS_Is_On (Sat 11-Jun-22 13:21:48)

Okay! Awaiting completion of build for first 10, actually 8.5, Gbit.

Once that's done will purchase the 2116. Existing equipment can comfortably handle a single circuit.

In reply to a post by Pheasant:

Looks lonely. Needs a mate 🤣

Not only did it get a mate but as you might've noticed it's a half-height card. I've switched out the brackets for full-height so that I can plug into a PC and tower server. The tower is served by a 25G DAC, the PC by a 25GBase-BX run.

So either way I've 25G in the property. Definitely no need to upgrade anything major for a while now.

Cool!

Did you get your 25G BX optics out of FS ok? They’re running a bit behind on some gear still.

In reply to a post by Pheasant:

Cool!

Did you get your 25G BX optics out of FS ok? They’re running a bit behind on some gear still.

Yep. Delivery is due mid-July.

The server next to the routers has a 25G DAC already connected.

The 25G router is currently offline.

The 2116 is waiting on confirmation that delivery of multiple circuits is possible. There's some missing duct meaning that rather than being connected to the underground DP just outside our shared drive we appear to end up somewhere else entirely.

No-one in their right mind would want to install the missing duct in block paving and a shared drive on a private road for the sake of 2 premises passed.

So I fully expect CityFibre to do so when they rock up in the next months.

Being serious we are on CF's plans but unless they're going to use underground infrastructure, no cabinets, they have no chance of being allowed to install anything. There are few places for cabinets to actually go and none that wouldn't involve digging in private roads.

In reply to a post by Pheasant:

Cool!

Did you get your 25G BX optics out of FS ok? They’re running a bit behind on some gear still.

25G BiDi SFP28s delivered yesterday, once customs charge had been paid: thanks Brexit!

That's the DAC and those here so the 2 x 25G legs are now ready to go.

Waiting for the node outside the property to be cabled and lit and we're good to go - the splicing was completed to the level 2 node last week, I'm to be served from a level 3 a couple of hundred metres of 48 fibre cable from the L2.

Nice got mine through too. With nice customs bill courtesy of DHL

Managed to add a couple of their GPON sticks to the order. These are all programmable using simple SSH command shell. Not bad for £50.

First connection is now doing its' thing. Will certainly need the higher power 2116 router I mentioned for any second circuit as this guy maxes out at about 15 Gb/s - exactly as init7 saw when testing their 25G service.

It's working fairly well - a small discrepancy in upload that's being worked through.

Moving on I did indeed purchase the 2116 to give me extra overhead. The next 'networky' purchase is going to be an insanely expensive switch with 25G+ ports and can wait for the foreseeable.

My masterplan is to have the 2116 as a router on a stick with its 4 x 10G ports in a LAG to the switch, using VLANs to separate WAN and LAN.

Obviously this means that the router has a maximum throughput of 20G up and down simultaneously as data hairpins through it however I can't see a scenario where this is inadequate for the foreseeable.

In reply to a post by XGS_Is_On:

No-one in their right mind would want to install the missing duct in block paving and a shared drive on a private road for the sake of 2 premises passed.

So I fully expect CityFibre to do so when they rock up in the next months.

Being serious we are on CF's plans but unless they're going to use underground infrastructure, no cabinets, they have no chance of being allowed to install anything. There are few places for cabinets to actually go and none that wouldn't involve digging in private roads.

CityFibre built a cabinet just outside the development and have used PIA throughout from there. Expect to see them become available imminently, however have no use for them right now. They'll only be providing GPON and even though it's symmetrical they'd only be a backup link and I've a special arrangement with my current backup provider.

That and I'd potentially have to use Vodafone. I've no interest at all in using Vodafone.

Sounds good.

Was on the wait for a pair of 2216’s from Getic/EuroDK since their intro date. They finally came into existence at the end of October and the price was far better than I could get here, so pushed the button.

Have put one into service so far, migrating from a Tile-based CCR running v6 onto this box running v7- had some migration grumbles to sort, but all good now. IPSec performance seems incredible by comparison to the Tilera.

In reply to a post by Pheasant:

Sounds good.

Was on the wait for a pair of 2216’s from Getic/EuroDK since their intro date. They finally came into existence at the end of October and the price was far better than I could get here, so pushed the button.

Have put one into service so far, migrating from a Tile-based CCR running v6 onto this box running v7- had some migration grumbles to sort, but all good now. IPSec performance seems incredible by comparison to the Tilera.

Yep! I know plenty that use them as LNS to terminate broadband customers so much the same workload.

Some muppet 🙈 here forget to check the depth of the 2216's compared to the older CCRs which are precisely 1/2 the depth in comparison. Need at least 450mm deep mounting. Result: deeper wall mount cab needed. Wait time 8-12 weeks for a deeper NetShelter 12U from APC/Schneider.

What switches are you considering?

Was looking one of the new Juniper EX4100's which look v.nice, but cant get stock for 25 weeks. Might go Cisco. Using a 2U Netgear M4300 elsewhere, but huge and overkill for here. Am not really a huge fan of Mikrotik switches but got a few tiny ones scattered about.

Wait times on everything at the moment still insane. Those 2216s took the best part of 8 months to begin shipping.

Have you got an overview sketch of the network you're planning on building?

I'm actually not considering any for right now and won't until I have to.

I have a switch that'll tick all the boxes from $employer however it's too deep for the cabinet where the rest of the kit lives. Cabinet is for 300 mm deep kit, this guy is just a touch deeper.

I'm going to hang back and wait for more miniaturisation. I don't actually need many ports as I have plenty of kit to provide more density.

This guy https://mikrotik.com/product/crs326_24s_2q_rm would be fine if it had some 25G ports but unfortunately it doesn't. Could have used a QSFP to 4 * SFP+ breakout cable.

When the time comes I'll have a look and see what's availability with QSFP and SFP28 ports. One thing I need to look at is how packets are being divided between the ports on the router and what the switch will support in hardware. Ideally I would like hardware accelerated layer 4 hashing so that a single host can max everything out.

There's no rush. XGSPON isn't going to be upgraded for a couple of years. I'm looking for some niche products based on 25GPON or 50GPON to arrive in the middle of the decade. The kit I have works for 2 * 10 G if it becomes a requirement.

This post sound very much the same to what i am in the process of building out. I have been checking out (over on ebay) some Brocade ICX6610 QSFP+ 40GbE switches for homelab and to learn more about routing/switching BGP/MLPS whilst using one as part of my homenet. Over on serverthehome forum theres a method to "fix" the device and in that discussion to disable the cap placed on these brocade switches to allow for extra proformance. You can still get ahold of these beasts for a couple of hundred bucks a pop. Apparently they are a bit on the loud side when running at full tilt but again these is a guide to mod the fans too, if im not mistaken. Also, if already failure with Cisco CLI supposedly these Brocade ICX6610's are almost identical from the CLI. For Those curious Brocade ICX on STH

With some breakout cables and adapters https://mikrotik.com/product/crs504_4xq_in would work however I'm going to sit back for now.

The ideal would be something with a few SFP28 ports and a couple of QSFP28 ports that ideally doesn't cost a grand and a half.

https://mikrotik.com/product/crs518_16xs_2xq would work but I really can't see an ONT with a QSFP28 port arriving any time in the next few years even when 50GPON becomes a commercial thing.

See where we are then. Hopefully supply will be way less constrained.

Serve the home have reviewed both of these switches, they both look pretty good.

CRS504

CRS518

Thanks Dan

Speaking with a couple of operators and looking at my own port utilisation stats on my kit it was about time for an upgrade to the main switch.

Rather than stay with Mikrotik given my pretty basic needs I've plumbed in a QNAP QSW-M5216-1T.

It's happily purring away with a couple of SFP28 DACs and a 25GBase-BX optical link.

Next upgrades will be lower port count switches with SFP28 ports and, probably about 2025, a router with a couple of QSFP28 ports, 1 of which will connect to the switch via a breakout cable.

Time to sell a few bits of old kit now, make back some of the cost of the QNAP 😬.

I found my M5216T runs surprisingly quite warm, even with mostly empty cages. PSU is internal though which is quite tight for the form factor so that may be it. Happily runs without complaint with FS supplied BiDi 25G optics

I've found the switch is a PoS incapable of even basic LAG functionality built by a vendor completely indifferent to supporting it