Second time today:

AFFECTING
Z.Witless
STARTED
Jan 26, 10:30 PM
https://aastatus.net/42613

Z.Witless hardware locked-up at 10:30 this evening, causing those line on it to drop an reconnect.


I am getting a bit sick of all these drops, could I get out of my remaining 10 months contract do you think?

It is unfortunate you seem to be on Z.Witless which appears to be affected more so than the other two LNS's, although that could just be a quirk of randomness and then us seeing a pattern that gets disproved over time. It's also bad luck you've joined just when these problems started and so have not known anything better.

I would suggest getting in touch with their technical support and raising a ticket, nothing will happen otherwise. I'm not sure 'legally' this would be enough to get you out of the contract, as all services can have problems and time needs to be allowed for companies to rectify them plus the issue is a blip rather than a long lasting outage, and these services come with no SLAs or guarantees of up time. Whether as a goodwill gesture they would let you leave early is of course only something they can tell you.

It has been a bit disappointing of late, but A&A are probably more disappointed than we are, and at least we know what is happening and are kept up to date.

Just had another unexpected outage, I had to reboot my router.
My connection now terminates with u.gormless.thn.aa.net.uk which unfortunately seems very laggy.

Just had another outage, when it was back up I logged into the control pages.

A pin has been added: "LNS Kill requested by andrew"
The status page https://aastatus.net/42617 explains:

INITIAL
3¼ hours ago by Andrew
At 14:00 The Y.Witless LNS locked up, causing customers connected to it drop and reconnect a few minutes later.

UPDATE
31 minutes ago by Andrew
Most lines reconnected by 14:11

UPDATE
29¼ minutes ago by Andrew
Some customers had re-connected to the "U.Gormless" LNS - which doesn't have as much throughput capacity as the Witless LNSs - in order to ease congestion we will manually force some customers to move off U.Gormless by way of a PPP kill - this will force the customer's router to reconnect causing a short outage (typically less than a minute)

RESOLUTION
6½ minutes ago by Andrew
The lock-up of Y.Witless was unfortunate as it did cause a disruption to some our customers this afternoon and we had hoped that the work done a week ago to Y.Witless would have helped prevent this hang. However, Y.Witless is out of service and in it's locked state, where our developers can connect to its CPUs and see if they can gain more information.


I am now back on z.witless.thn.aa.net.uk

Edited by perlen (Sat 03-Feb-24 17:29:51)

I'd probably be off at this point. A nice idea to run your own hardware but it seems like it's not working as planned.

And it goes down again!
Right in the middle of a Teams meeting the Mrs was in (WFH)


REFERENCE
42618 / AA42618
PERMALINK
https://aastatus.net/42618
INFORMATION
At around 17:20 the Z.Witless LNS hung, causing customers on it to drop and reconnect.

Edited by perlen (Mon 05-Feb-24 17:37:44)

Sorry Perlen, and others affected by this.

The post https://aastatus.net/42608 has more information about the problem.

Due to the work we did last week we have been able to gain more information about today's (and Saturday's) lockup than we have been able to in the past. This low-level data comes from the CPU, memory and other hardware on the system whilst it's in the 'hung' state. This is being analysed and is providing clues, but work analysing this is still ongoing.

"We are not only an Internet Service Provider. We also design and build our own routers under the FireBrick brand."

I do wish you success in fixing this. Although I'm not an AAISP customer, it does sound like the brand risks becoming badly tainted by this saga.

I wonder if the older generation Firebricks are still around? Could these be used for live customer traffic, whilst the newer generation are used with an opt-in pool of beta testers?

In reply to a post by candlerb:

I wonder if the older generation Firebricks are still around? Could these be used for live customer traffic, whilst the newer generation are used with an opt-in pool of beta testers?

As I understand it the older generation Firebricks are still in use but for slower customers (<80Meg). When I first joined I found myself connecting to the older LNS's (Gormless) even though I was a 1000/100 customer. How did I know? Speeds were only up to around 300Meg. This was resolved very quickly by contacting them and a reconnect saw me on the new LNS's and speeds as expected. So it seems to me the older kit struggles with faster connections and so can't be used as a fallback for faster services.

They are now upgrading some of the older LNS's to the newer ones, they said the idea being a drop of one would not affect as many customers. I'm not sure that logic works though as if the LNS's are all equally as prone to locking up, then it doesn't matter how many of them there are, lockups will affect the same number of customers, just in smaller batches over more bits of kit. Perhaps they need the extra capacity as while they debug locked up boxes they remain out of use.

I'm sure it will all be sorted out soon now they have some debug data from the locked up boxes. Be interesting to find out the cause.

Edited by E300 (Tue 06-Feb-24 08:44:09)

The older generation firebricks have a 1Gb backplane and core connection, hence no good for the faster connections, whereas the latest firebrick 9000 has a 10Gb backplane and core connection. What I find puzzling is that of the 3 original witless 9000,s (X,Y,Z), whilst Y and Z have locked up fairly regularly, X (which I happen to be connected to) has an uptime of around 82 days. Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?