In reply to a post by bellerby:

The older generation firebricks have a 1Gb backplane and core connection, hence no good for the faster connections, whereas the latest firebrick 9000 has a 10Gb backplane and core connection. What I find puzzling is that of the 3 original witless 9000,s (X,Y,Z), whilst Y and Z have locked up fairly regularly, X (which I happen to be connected to) has an uptime of around 82 days. Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?

I'm now on X as well which also gets me the lowest latency, hopefully there I remain

Yes I would think if all else is the same it seems to suggest some low level hardware issue, which could be anything of course. I'm sure they've compared the differences between X and the others to try and pinpoint what might be up. X might well have the same issue, but differences in component tolerances might just be enough to keep it far enough away from whatever 'cliff edge' the other hardware falls over.

I'm glad I'm not the person having to sort it out!

They have recently moved customers off of gormless a, b & c with the intention of replacing these with more FB9000's tomorrow.

https://aastatus.net/42616

In reply to a post by bellerby:

Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?

As someone who manufactures Pcb's on a daily basis, I think it's unlikely to be a pcb build issue.
The Firebricks are manufactured on a very small scale so they won't do small test samples from batches. The pcb manufacturer will fully test each board including AOI (automatic optical inspection) and a whole suite of electrical testing (including things like flying probe testing).
Most hardware issues come from components added to the finished pcb.

The Firebrick PCB is nowhere near as complex as some of the other circuit boards made today (smaller tracks, tighter spacing, many more layers, thousands more tiny via's between layers).

I have no idea where Firebrick have their PCB's made but I know they are made in the UK.
Pretty much all UK based PCB manufacturer is low volume, high precision, quick turnaround work. Anything with high volume gets made overseas.

I would put my money on it being a software issue causing them to hang or an issue with a component on the board (chip/ram etc).

In reply to a post by j0hn83:

They have recently moved customers off of gormless a, b & c with the intention of replacing these with more FB9000's tomorrow.

https://aastatus.net/42616

Let me check I understand this. They're moving all remaining customers off the original stable and reliable LNSes, onto the new ones which have a crash rate of 2 in 3? Before they've fully diagnosed the problem, which they can't reproduce in the lab, but only happens when live customers are put on them?

In reply to a post by candlerb:

In reply to a post by j0hn83:

They have recently moved customers off of gormless a, b & c with the intention of replacing these with more FB9000's tomorrow.

https://aastatus.net/42616

Let me check I understand this. They're moving all remaining customers off the original stable and reliable LNSes, onto the new ones which have a crash rate of 2 in 3? Before they've fully diagnosed the problem, which they can't reproduce in the lab, but only happens when live customers are put on them?

That's certainly how I'm reading it.

Beta testing is usually voluntary but I guess not always.

Edit: don't think it's all customers, but certainly 3 Firebricks worth

On the status page linked further up by Andrew they write

The enlarged pool of LNSs will also reduce the number of customers affected if there is a lock-up of one LNS.

That only makes sense ifthey deploy more FB9000's to the current customers on those devices. Throwing in more customers from the older, stable Firebricks will just mean those customers are now more likely to be affected.

Why not add the extra FB9000's, spreading out the witless LNS load, but leave the stable gormless LNS's as they are? Probably down to the cost of rack space.

I don't intend on lecturing A&A on their network 😂 they are smarter than me. It just seems a bit backwards. Credit for the transparency though.

Edited by j0hn83 (Tue 06-Feb-24 13:24:39)

In reply to a post by candlerb:

Let me check I understand this. They're moving all remaining customers off the original stable and reliable LNSes, onto the new ones which have a crash rate of 2 in 3? Before they've fully diagnosed the problem, which they can't reproduce in the lab, but only happens when live customers are put on them?

As I understand it, they are decommissioning several Gormless LNSs (on the stable hardware) and moving the customers over to other Gormless LNSs still on stable hardware, as they said they have plenty of capacity for slower customers (<80Meg).

I suspect they are doing this to free up rack space at the datacentre to then install a few more newer LNSs. Only customers already needing to connect to the newer LNSs (>80Meg) will connect to these upgraded ones. So no extra customers are using the troublesome new LNSs than they were before. The idea is if an LNS drops, then as there are fewer customers on each LNS then not so many people are affected by a single LNS crash.

I guess you could argue is it a good idea to add more problematic kit into the mix. Also if the new LNSs have the same probability of a crash, you aren't having less customers affected, just smaller batches of customer drop more often.

Edited by E300 (Tue 06-Feb-24 13:20:26)

I guess another possibility is that the issue could be load related. The FB9000 was first introduced into the AA network in early 2022. I don't have any recollection of these lock ups being an issue during the first 12 months or so. Since early 2022 I would assume that a lot of the installed base would have migrated from ADSL/VDSL to FTTP. This is probably why A&A are now oversubscribed with the older FBs, and no doubt the load being placed on the 9000s will be inexorably increasing, both in number of connections and total throughput. Possibly the move to expand the pool of 9000s is to rule out this possibility.

Please excuse my intrusion into others' grief, but I have a simple question that may be completely unrelated. I have Fibre BB from another supplier and have no intention of moving it to A&A. However I am considering moving just my current DV line to A&A. Is any of the current trouble likely to have any impact on my phone connection?

No, it will be completely affected. The equipment being discussed, known as LNS or BNG, is what terminates broadband connections coming into their network. The traffic between their voice servers and the Internet won't go through these.

In reply to a post by E300:

As I understand it, they are decommissioning several Gormless LNSs (on the stable hardware) and moving the customers over to other Gormless LNSs still on stable hardware, as they said they have plenty of capacity for slower customers (<80Meg).

This is correct.

We have 20 or so of the FireBrick FB6000 LNSs which have more than enough capacity for the lower speed customers we have.

Once we've replaced a, b, c, d Gormless with FireBrick FB9000s they will then be put in to service for the faster-speed customers.

This isn't because the existing LNSs are overloaded (we don't believe load is the cause of the lock-ups), but to spread the load so fewer customers are affected.

Happy to answer more question if there are any!