In reply to a post by E300:

In reply to a post by bellerby:

So possibly BTW/TTB connections are on the 4 gormless LNS with City Fibre on the 3 witless lns.

I did wonder as well if they thought the City Fibre traffic was somehow causing a bug, hence the separation of traffic. One thing seems clear, given X.Witless had the longest uptime out of all [...]

Counterexample: Openreach FTTP here, on x.witless until yesterday, then reconnected to z.witless.
Crash messes up BQM, too, with history from preceding midnight to crash time becoming inaccessible. Reliability and BQM are two unique selling points of AA, and both are negatively affected by these crashes. I'm sure they are aware of this, but IMNSHO the steps taken to fix this so far haven't worked, therefore different steps would be now be advisable.

In reply to a post by qazwsxedc:

Counterexample: Openreach FTTP here, on x.witless until yesterday, then reconnected to z.witless.
Crash messes up BQM, too, with history from preceding midnight to crash time becoming inaccessible. Reliability and BQM are two unique selling points of AA, and both are negatively affected by these crashes. I'm sure they are aware of this, but IMNSHO the steps taken to fix this so far haven't worked, therefore different steps would be now be advisable.

So in that case, that blows that theory that City Fibre connections might somehow be triggering a bug if that was what they were thinking as X.Witless appears to be an Oprenreach LNS.

Seems to be a lack of updates currently, presumably because they are no further forward.

Goodbye x.witless, thank you for your service:


Mar 15, 07:40 AM
https://aastatus.net/42643

1½ hours ago by Andrew
At 7:30AM, the X.Witless restarted causing customers on it to drop and reconnect.

UPDATE
1 hour ago by Andrew
Lines reconnected by 7:33

RESOLUTION
1 hour ago by Andrew
This was related to https://aastatus.net/42608 This LNS is now out of service and will be analysed by our developers.

The irony is X.Witless was the most reliable LNS, they then put an NVMe drive in it like the others as they suspected the slot left empty was causing some instability, now X has been the least reliable just recently.

We've had no updates either for a while so that kind of points to them not being any further forward. I really want to be hearing about plan B's now as this has gone on for months.

Seems it was a rough night over night as awoke to a string of emails of link up and down, turns out due to BT testing some links https://aastatus.net/42648 Seems pretty poor of BT to do that kind of testing with no notice. Found myself on t.gormless and latency up to 9.5ms from 7.5ms the day before which was g.gormless I think, I did a drop and reconnect this morning and back on x.witless and latency at 6.5ms which feels a lot nicer . For some reason only x.witless gets my latency that low.

It also seems at some point this week new firmware is going up to the LNS boxes that may contain a fix for the lockups, currently the LNS boxes are running on older but stable firmware hence we've not had any crashes recently, that's according to the latest update at https://aastatus.net/42647 The status note isn't clear but it seems to imply they are all being updated, I would hope they are just updating one or two first in case it makes things worse and not better. Fingers crossed it fixes things.

Edited by E300 (Tue 26-Mar-24 10:22:04)

I think they were doing LNS full upgrades last night according to the updates on https://aastatus.net/42647

The last two updates were:

UPDATE
1 day ago by Andrew
These upgrades are now scheduled to happen this week (Tues 26th through to Thursday 28th) from 3AM.

UPDATE
1¾ hours ago by Andrew
Due to BT Planned work on one of our hostlinks, https://aastatus.net/42640 we will not be scheduling any upgrades for the early hours of Wednesday 27th March.

So I think they are only avoiding doing the upgrades this evening, but were scheduled to do them last night and there may be further to come tomorrow night.

I guess this might have been disrupted by the BT link issue, but we won't know for certain until there is a further update on: https://aastatus.net/42647

Well we had drops yesterday (Sunday) which don't seem to get flagged up now on the service status page, that put me onto a gormless LNS due for an update which then meant I got knocked off again overnight! Currently sitting on y.witless.

Not sure what the current status is for the latest fixes as the information seems a bit fragmented on the status page, but I think I read it as they've had to roll back to the last firmware that was stable, so haven't yet found the cause.

Yes, 10am Sunday I lost connection too.
No explanation from AAISP:


https://aastatus.net/recent.cgi

Another drop last night before 1am. Not sure what the cause was as nothing on the service status page. It seems drops are becoming the norm and so don't get a mention anymore on the service status pages.

It's been about 5 months since the problems started and no closer to a fix it seems. We keep going through cycles of upgrades then downgrades all of which cause more drops on top of the random crashes.

I would suggest they move to just having one live LNS they test the fix on for the crashes and keep the rest on the stable firmware, and allow their customers to choose to be BETA testers and connect to this test LNS. In order to get a decent number on it for a real world test, as an incentive give people a discount to become a BETA tester. That way they could also update this LNS more frequently and during more sociable hours, as I bet the person having to do these updates and downgrades are fed up with doing it in the dead of night.

Edited by E300 (Thu 11-Apr-24 08:33:57)

For a company that tries to pride its self on transparency the ball really has dropped recently. The FireBrick platform has some good features but the trade off is an unstable internet connection at a premium cost. I’m rooting for the team to get things fixed but they might have to go back to the drawing board and admit the current platform isn’t going to work for customers. Noticeable speed drops, connection drops and lack of transparency is making me heavily reconsider who I go with next. It won’t be A&A.