Poor uptime and reliability

Edited by perlen (Wed 10-Jan-24 23:17:51)

Edited by E300 (Thu 11-Jan-24 09:33:48)

Edited by Rhynchelma (Thu 11-Jan-24 09:56:10)

Edited by perlen (Thu 11-Jan-24 22:02:44)

Edited by XGS_Is_On (Fri 12-Jan-24 08:04:26)

In reply to a post by jimbof:

The spec always seemed a bit odd to me (only 2x 10G ports) given that Openreach are now selling >Gbit services.

In reply to a post by perlen:

Z.Witless is now in service and on new hardware - I was moved back on to it at 16:33 today.

Unfortunately base latency (was 9ms now 11ms) has increased by 2ms

In reply to a post by jpm:

Have they explained why they are doing this sort of work at 5pm instead of scheduling overnight changes?

Edited by E300 (Fri 12-Jan-24 18:32:58)

In reply to a post by perlen:

And another:

Jan 26, 07:00 AM
https://aastatus.net/42612

6AM: Z.Witless LNS had a hardware lock-up, causing lines on it to drop and reconnect

Edited by perlen (Sat 03-Feb-24 17:29:51)

Edited by perlen (Mon 05-Feb-24 17:37:44)

In reply to a post by candlerb:

I wonder if the older generation Firebricks are still around? Could these be used for live customer traffic, whilst the newer generation are used with an opt-in pool of beta testers?

Edited by E300 (Tue 06-Feb-24 08:44:09)

In reply to a post by bellerby:

The older generation firebricks have a 1Gb backplane and core connection, hence no good for the faster connections, whereas the latest firebrick 9000 has a 10Gb backplane and core connection. What I find puzzling is that of the 3 original witless 9000,s (X,Y,Z), whilst Y and Z have locked up fairly regularly, X (which I happen to be connected to) has an uptime of around 82 days. Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?

In reply to a post by bellerby:

Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?

In reply to a post by j0hn83:

They have recently moved customers off of gormless a, b & c with the intention of replacing these with more FB9000's tomorrow.

https://aastatus.net/42616

In reply to a post by candlerb:

In reply to a post by j0hn83:

They have recently moved customers off of gormless a, b & c with the intention of replacing these with more FB9000's tomorrow.

https://aastatus.net/42616

Let me check I understand this. They're moving all remaining customers off the original stable and reliable LNSes, onto the new ones which have a crash rate of 2 in 3? Before they've fully diagnosed the problem, which they can't reproduce in the lab, but only happens when live customers are put on them?

The enlarged pool of LNSs will also reduce the number of customers affected if there is a lock-up of one LNS.

Edited by j0hn83 (Tue 06-Feb-24 13:24:39)

In reply to a post by candlerb:

Let me check I understand this. They're moving all remaining customers off the original stable and reliable LNSes, onto the new ones which have a crash rate of 2 in 3? Before they've fully diagnosed the problem, which they can't reproduce in the lab, but only happens when live customers are put on them?

Edited by E300 (Tue 06-Feb-24 13:20:26)

In reply to a post by E300:

As I understand it, they are decommissioning several Gormless LNSs (on the stable hardware) and moving the customers over to other Gormless LNSs still on stable hardware, as they said they have plenty of capacity for slower customers (<80Meg).

In reply to a post by candlerb:

No, it will be completely affected. The equipment being discussed, known as LNS or BNG, is what terminates broadband connections coming into their network. The traffic between their voice servers and the Internet won't go through these.

In reply to a post by andrewhearn:

This is correct.

We have 20 or so of the FireBrick FB6000 LNSs which have more than enough capacity for the lower speed customers we have.

Once we've replaced a, b, c, d Gormless with FireBrick FB9000s they will then be put in to service for the faster-speed customers.

This isn't because the existing LNSs are overloaded (we don't believe load is the cause of the lock-ups), but to spread the load so fewer customers are affected.

Happy to answer more question if there are any!

In reply to a post by andrewhearn:

I think you mean: No, it will be completely unaffected

In reply to a post by j0hn83:

In reply to a post by bellerby:

Assuming they are all running the same software/firmware then does this imply some very low down hardware/component/pcb build issue?

...snip...
I would put my money on it being a software issue causing them to hang or an issue with a component on the board (chip/ram etc).

Edited by perlen (Tue 27-Feb-24 12:35:37)

Edited by E300 (Tue 27-Feb-24 13:27:15)

In reply to a post by E300:

I wonder if they have a plan B, for example buying in some alternative hardware?

In reply to a post by jalzoo:

Was with AAISP for 3 months via City fibre and I can honestly say it was the most unreliable expensive over rated ISP I have ever been with. I changed over to IDNet and I have had absolutely no problems at all plus I'm saving money happy days.

In reply to a post by jalzoo:

It might be cheaper and more stable but you don't have Continuous Quality Monitoring anymore.

Edited by jalzoo (Wed 06-Mar-24 15:11:13)

In reply to a post by bellerby:

Indeed not. However they have split off City Fibre connections from the rest. I'm on BTW and currently connected to g.gormless. So possibly BTW/TTB connections are on the 4 gormless LNS with City Fibre on the 3 witless lns. Further conjecture is the thought that maybe the issue is triggered by CF. Looking at the history, the issue does appear to have started afer taking on CF. Just a thought.

In reply to a post by E300:

In reply to a post by bellerby:

So possibly BTW/TTB connections are on the 4 gormless LNS with City Fibre on the 3 witless lns.

I did wonder as well if they thought the City Fibre traffic was somehow causing a bug, hence the separation of traffic. One thing seems clear, given X.Witless had the longest uptime out of all [...]

In reply to a post by qazwsxedc:

Counterexample: Openreach FTTP here, on x.witless until yesterday, then reconnected to z.witless.
Crash messes up BQM, too, with history from preceding midnight to crash time becoming inaccessible. Reliability and BQM are two unique selling points of AA, and both are negatively affected by these crashes. I'm sure they are aware of this, but IMNSHO the steps taken to fix this so far haven't worked, therefore different steps would be now be advisable.

Edited by E300 (Tue 26-Mar-24 10:22:04)

Edited by E300 (Thu 11-Apr-24 08:33:57)

In reply to a post by Sun4Lw5LIQy:

For a company that tries to pride its self on transparency the ball really has dropped recently. The FireBrick platform has some good features but the trade off is an unstable internet connection at a premium cost. I’m rooting for the team to get things fixed but they might have to go back to the drawing board and admit the current platform isn’t going to work for customers. Noticeable speed drops, connection drops and lack of transparency is making me heavily reconsider who I go with next. It won’t be A&A.

Edited by E300 (Thu 11-Apr-24 18:03:07)

In reply to a post by candlerb:

The decision to remove the older, slower but reliable LNSes (which were providing service to the customers on lower speed connections), and replace them with a model which is faster but known to be unstable, was madness IMO.

In reply to a post by j0hn83:

but it added additional FB9000's to spread the witless LNS load.

Edited by E300 (Fri 12-Apr-24 14:48:23)

In reply to a post by j0hn83:

Those on the slower, more reliable FB6000 LNS's (nicknamed gormless I believe) were moved to other FB6000's. The FB6000's were swapped with the newer troublesome FB9000's (nicknamed witless).
So nobody was moved from an FB6000 to an FB9000, but it added additional FB9000's to spread the witless LNS load.

Edited by aabloor (Fri 12-Apr-24 16:47:38)

I doubt any other ISP develops its own core equipment.

Edited by E300 (Fri 12-Apr-24 17:38:49)

In reply to a post by Rhynchelma:

Are all of these dropouts the fault of A&A? Are any of them Openreach or who ever?

Edited by E300 (Fri 12-Apr-24 17:59:48)

In reply to a post by E300:

In reply to a post by Rhynchelma:

Are all of these dropouts the fault of A&A? Are any of them Openreach or who ever?

Some recent drops were a problem created by suppliers, they typically happen overnight.

The problem we are discussing here has been an issue since November last year, where randomly and often during the day the connection has dropped for a quantity of customers because of a hardware problem with A&A's Firebrick they design themselves.

Apart from the recent BT issue that caused a lot of drops over a single night, I don't recall any other drop that wasn't caused by A&A and this issue, or due to A&A having to upgrade or downgrade firmware in order to try and fix it. To put things into perspective, the drops were not very often, I perhaps had one every few weeks, but less so now as they've reverted to stable firmware.

In reply to a post by jimbof:

Pretty limited utility to the "average" punter, though. The bandwidth accounting features are useful to AAISP with their commercial model, but not really to the customer, other ISPs simply don't charge for bandwidth. Much of the functionality from the CQM side can be achieved using 3rd party services (arguably even more useful, as it takes into account issues in the ISP's transit).And the bonding services must be dying a death now for residential, with rubbish ADSL lines rapidly being something folk don't need to contend with, though maybe more useful to businesses for failover (I think central bonding is a bad idea for that use case, though, better using distinct network providers with local failover).

I raised the question of the stability of the (then beta) LNS HW around a couple of years ago when I joined AAISP, I was a little concerned it could be an issue. As it would happen, over the period I was a customer the LNS behaved impeccably; I left because no matter the quality of service, the anxiety over the quota system and trying to juggle changing quotas to use up my allowance at minimum cost just annoyed me too much.

I'm with Unchained now, who use Cisco for their LNS; performance is great (900Mbps line rate single threads to Cloudvider iperf services in UK and often EU) - and no quotas at lower cost than the least expensive AAISP service. Small outfit, great personal service. I certainly don't miss any features from the FB9000 I'm (no longer) connected to. I think it is a tenuous benefit for many, and in hindsight my concern was probably founded in common sense (and I got lucky over my tenure).

Edited by E300 (Sun 14-Apr-24 09:37:21)

In reply to a post by aabloor:

We do still believe strongly that the FB9000, when stable, offers us features that distinguish our service from the service of almost all others. Simply, we want bonding, CQM graphs, low power consumption, etc.

It is part of what makes our ISP offering different and better; our USP.

In reply to a post by aabloor:

This is where our two roles; that of both an ISP with broadband customers, and also that of a hardware manufacturer meet each other head-on and, unfortunately and uncomfortably, collide.

Edited by XGS_Is_On (Sun 14-Apr-24 12:37:26)

In reply to a post by jpm:

A selling point of A&A is the L2TP service that you can use if your primary service goes down. I don't know how easy it would be to provide that sort of functionality if you remove the "session" element of PPPoE and resort to DHCP. Maybe the easiest way to achieve it would be to have the L2TP and FTTP service allocate different IP addresses and relying on the end user to run a routing protocol to use their subnet on whatever connection they decided they wanted it on.

In reply to a post by XGS_Is_On:

Another question related is are you folks going to use PPP indefinitely given a major feature relies on it? Openreach deliver Ethernet, CityFibre deliver Ethernet, at some point BT Wholesale will deliver Ethernet, they wanted to move away from PPP when GEA started, TTW etc will follow. Are you folks going to be having the CPE putting customer data in PPP purely for the LCP echoes to keep CQM running?

Lower power consumption is super important but per subscriber how are the numbers?

In reply to a post by E300:

That is one of the things I was hoping A&A might have had, the option to use IPoE to avoid needing an over powered pfSense or OPNsense box to get the 1Gig throughput, due to PPP having to be done in software. It's the sort of thing they should be doing to keep the more technical customer happy

In reply to a post by E300:

That is one of the things I was hoping A&A might have had, the option to use IPoE to avoid needing an over powered pfSense or OPNsense box to get the 1Gig throughput, due to PPP having to be done in software. It's the sort of thing they should be doing to keep the more technical customer happy

In reply to a post by candlerb:

The more technical customers (and those prepared to pay A&A rates) will also likely use decent hardware and/or software.

PPPoE at 1Gbps is not hard to achieve in software. It's only an 8-byte extra header; similar work to VLAN tagging (4 bytes).

In reply to a post by XGS_Is_On:

Think hardware accelerated PPPoE has been in SoCs for a while now, the weaksauce consumer routers ISPs give need it to hit the throughput target. Shouldn't need too much to get to gigabit throughput through software though unless using the software you described which uses a kernel with bad PPP functionality: single core decapsulation only, no multithreading.

BTW would have to make significant change to their network to change this, it's not something an ISP can implement on their own.

Edited by E300 (Mon 15-Apr-24 08:51:05)

In reply to a post by jimbof:

There are a couple of holdouts where companies are not doing HW accelerated PPP on hardware that is barely capable of doing it at line rate; Ubiquiti being one obvious example. I'd welcome a DHCP / Ethernet based service as I do use Ubiquiti gear and have put an OPNsense box in front of it to do PPP...

This seems like a moot discussion though; the only companies that have done DHCP based FTTx in the UK are those who have their own equipment in the exchanges - so Sky and TalkTalk retail I believe. All the BTW backhaul based consumer services like AAISP are PPP because that is how that service works; the login details used steer the connection to the target ISPs LNS. BTW would have to make significant change to their network to change this, it's not something an ISP can implement on their own.

Edited by XGS_Is_On (Mon 15-Apr-24 10:10:08)

In reply to a post by E300:

Guess that explains it then. Maybe they have the option with CityFibre in the future?

In reply to a post by XGS_Is_On:

They do, but until BTW start changing it is it worth it because a small subset of users have marginal hardware running single threaded PPP? All expense for zero practical gain. Need more of a business case, especially when your network is engineered around PPP.

Edited by E300 (Mon 15-Apr-24 10:38:14)

No requirement to use PPP to steer traffic to BTW customers: it's not dialup and they know which port on the DSLAM/OLT the connection came in on which is all they need. Map that to the ISP, repackage in different VLANs, EVPN, off it goes.

In reply to a post by E300:

A&A do try to do things differently, try to be ahead of the game

Talktalk only did IPoE because they needed multicast for their old TV service. It's not because they want to make a service for techno-nerds. After all, they're *not* deploying IPv6 either.

In reply to a post by jimbof:

No requirement to use PPP to steer traffic to BTW customers: it's not dialup and they know which port on the DSLAM/OLT the connection came in on which is all they need. Map that to the ISP, repackage in different VLANs, EVPN, off it goes.

When you say "no requirement" - you're talking about a hypothetical network setup BT could deploy, right, not something that is available now but ISPs aren't choosing to use?

BTW /Openreach appear to be doing the bare minimum of anything with the FTTP implementation, and there looks to be no steering in place with the BTW ISPs I've used.

I have had 2 BTW connections at the same property on the same PON, and BTW don't even police that a certain ISP PPPoE login is originating from a certain ONT. I could happily connect to either BTW ISP (in this case, AAISP or Unchained) via either of the ONTs that I had, and the connection ended up at the ISP whose login details you used, with no obvious error if you used the wrong ONT. In the case of AAISP, it totally broke their CQM setup as their CQM setup looks for the incoming ID it is expecting to tie up the data with your account, yet they still allows the PPPoE connection.

In reply to a post by Sun4Lw5LIQy:

For a company that tries to pride its self on transparency the ball really has dropped recently. The FireBrick platform has some good features but the trade off is an unstable internet connection at a premium cost. I’m rooting for the team to get things fixed but they might have to go back to the drawing board and admit the current platform isn’t going to work for customers. Noticeable speed drops, connection drops and lack of transparency is making me heavily reconsider who I go with next. It won’t be A&A.

VM Gig1 - AAISP L2TP

In reply to a post by jpm:

A selling point of A&A is the L2TP service that you can use if your primary service goes down. I don't know how easy it would be to provide that sort of functionality if you remove the "session" element of PPPoE and resort to DHCP. Maybe the easiest way to achieve it would be to have the L2TP and FTTP service allocate different IP addresses and relying on the end user to run a routing protocol to use their subnet on whatever connection they decided they wanted it on.

VM Gig1 - AAISP L2TP

In reply to a post by Sun4Lw5LIQy:

I keep getting hopped over different LNS devices and it’s a bit of a lottery. I am paying for 1GB internet and I’m lucky if I get 330MB. Some days it’s 200. I had the same router and ONT with wholesale TalkTalk and I was getting 800MBs. It feels like a downgrade.

VM Gig1 - AAISP L2TP

In reply to a post by Sun4Lw5LIQy:

I have contacted support about the speed issues but have been told it’s my WiFi. Again same unit as my previous ISP. I suspect it’s due to BT Openreach being my carrier instead of TalkTalk Business which had LLU’d the local exchange. I suspect that’s how I’ve been able to get better connectivity in the past.

I was originally provisioned to the incorrect LNS by A&A they did fix it but I still get speed drops.

Edited by E300 (Wed 17-Apr-24 13:09:39)

In reply to a post by Sun4Lw5LIQy:

Just for clarity I also have experienced the LNS stalling issues I was told they are looking into it and they had added NVMe drives to some of the LNS systems to get things resolved.


I have contacted support about the speed issues but have been told it’s my WiFi. Again same unit as my previous ISP. I suspect it’s due to BT Openreach being my carrier instead of TalkTalk Business which had LLU’d the local exchange. I suspect that’s how I’ve been able to get better connectivity in the past. I was originally provisioned to the incorrect LNS by A&A they did fix it but I still get speed drops.

Like I said when it comes to contract renewal I’ll be looking elsewhere.

VM Gig1 - AAISP L2TP

Edited by Chrysalis (Wed 17-Apr-24 13:57:58)

In reply to a post by E300:

In reply to a post by Sun4Lw5LIQy:

I have contacted support about the speed issues but have been told it’s my WiFi. Again same unit as my previous ISP. I suspect it’s due to BT Openreach being my carrier instead of TalkTalk Business which had LLU’d the local exchange. I suspect that’s how I’ve been able to get better connectivity in the past.

You really need to test using a wired connection. Wi-Fi is fairly fluid, it can work great one day and not so the next depending on interference or if it has decided switch channels etc.

I was originally provisioned to the incorrect LNS by A&A they did fix it but I still get speed drops.

I had the same problem 2 years ago and provisioned on to the wrong LNS, I'd have thought they had got this fixed by now, obviously not.

You could try connecting to the test LNS and see if that gives you better speed, just prefix your username with test- and reconnect. I see better single thread speeds when on that one presumably because it isn't loaded up with many customers. If its better you can assume the slow downs are not due to BTWholesale and backhaul

VM Gig1 - AAISP L2TP

VM Gig1 - AAISP L2TP

Edited by Chrysalis (Wed 17-Apr-24 15:19:05)

In reply to a post by Chrysalis:

I was surprised at the resistance to utilising cisco as a stop gap, and no compensation got mentioned

Edited by E300 (Wed 17-Apr-24 16:02:05)

In reply to a post by jcre:

Whilst there's technical reasons to keep PPP in the BTW and TTB networks that would be harder if it was a pure layer 2 delivery (at least in my opinion), they both do offer VLAN based handover on their EoFTTx products - which attract a fair premium, and they probably want to keep EoFTTx as their premium offering to maintain that extra profit margin, which would incentivise keeping PPP in place for their less premium offerings.

FTTx is already handed over on the L2S in the exchange as a VLAN anyway.

Edited by XGS_Is_On (Wed 17-Apr-24 20:34:22)

In reply to a post by E300:

In reply to a post by Chrysalis:

I was surprised at the resistance to utilising cisco as a stop gap, and no compensation got mentioned

Exactly, I've not heard of a plan B except they haven't got one, they are just blinkered into Firebrick and that's it, and I just got to the point of thinking it is too much money a month to be a beta tester. I agree they should have offered some rebate or reduced monthly premium whilst these issues continued to those affected.

In reply to a post by perlen:

Good point E300, aabloor the General Manager of A&A registered here on Fri 12-Apr-24 10:24:36 to post his message:
https://forums.thinkbroadband.com/aaisp/t/4755788-re...

he finishes that message with:
"Thanks for taking the time to read this, and we are happy to answer any questions, of course."

But no response has been made to ANY of the questions that have followed... and he still has:
Total Posts 1

Edited by Chrysalis (Thu 18-Apr-24 13:43:51)

In reply to a post by Chrysalis:

Well I didnt say that, was surprised it wasnt considered for temporary use. Maybe a naive thought.

In reply to a post by aabloor:

Within this same time frame, we have had multiple instances of BT Wholesale doing planned work which they had not told us about in advance (and apparently not told other ISPs, too). We could have zeroed the impact of their planned work, had they told us they were doing it beforehand.

Multiple times we have raised this with our account manager and at higher levels, and we still have not had a satisfactory response. Of course, no wholesale network is 100% reliable; we are not unreasonable about this, but the combined appearance, especially to customers not following matters closely, is that it's "another LNS blip".

Edited by Sun4Lw5LIQy (Tue 23-Apr-24 14:05:30)

In reply to a post by Sun4Lw5LIQy:

More speed drops. Speed went to 100MB, after reboot it’s now sitting at 200MB. Paying for 800MB. Again before anyone jumps to conclusions I used the very same router and ONT with my previous provider and the speeds was around 800MB. It was business broadband wholesale talktalk previously. Now the back haul is over bt open reach and not using the LLU kit at the exchange.

Called up A&A. Put through to the sales line. Been told it will cost me £160 to cancel the service. Really don’t like being a beta tester for a product. Silence is golden from A&A on the forum says everything I need to know as a customer. I use the line as a fully remote worker so I depend on stable speedy internet and I’m very disappointed in the service.

Edited by aabloor (Tue 23-Apr-24 16:37:38)

In reply to a post by aabloor:

Tangential thought :

One possible use of AI would be to make a stab at interpretation of graph results.

We probably won't do it, but it feels like it would be a perfect application.

Alex.

In reply to a post by aabloor:

SOOC manufacturers tend to assume you will be running something like Linux on their chips. Certainly out approach of an entire ground up OS, hardware drivers, networking etc is rare to the extend that some initially have trouble believing it to be true. But it really is.

Edited by aabloor (Wed 24-Apr-24 11:39:57)

In reply to a post by aabloor:

It is an ambitious business model ...

Edited by Chrysalis (Sat 04-May-24 01:48:17)

In reply to a post by perlen:

I am so glad to no longer be a beta tester, and once again have the reliable connection that I pay for!

In reply to a post by jimbof:

I am aware, but it seems they've said part of their issue may have been not having enough folk on the test LNS before it went to live; there isn't any incentive at present to be on the test server (particularly given the premium pricing). Some kind of discount might encourage more to sign up for the test LNS.

In reply to a post by gorebrush:

Week 2 April here and it's been 100% other than when I've been faffing internally.

Edited by Chrysalis (Fri 24-May-24 16:35:30)

Our LNSs remain stable running the 'Factory' software. Work continues in our test lab and on non-customer affecting parts of our network to track down the problem with the alpha software.

For the past month or so our service has no incident of LNS hangs, most of our FB9000 LNSs have an uptime approaching 100 days. We are confident that the Factory software is stable.

In reply to a post by candlerb:

Update on the status page:

Our LNSs remain stable running the 'Factory' software. Work continues in our test lab and on non-customer affecting parts of our network to track down the problem with the alpha software.

For the past month or so our service has no incident of LNS hangs, most of our FB9000 LNSs have an uptime approaching 100 days. We are confident that the Factory software is stable.

Interesting that it was "alpha" software that was causing the problems. But good that the released software is stable.

In reply to a post by candlerb:

I do wonder though, what shiny new features were so important in this new software that (a) they put it straight into production, and (b) they persisted with trying to fix it *in production* for months, rather than rolling back to the stable software straight away.

The positive from this is that the existence of software which is stable under the same load strongly suggests that it's *not* flaky hardware after all. Not 100%, but very likely.

Edited by E300 (Sat 25-May-24 09:50:00)

In reply to a post by E300:

I'm assuming they've gone back to the original Firebrick OS that only runs on 2 cores and the OS they need to get working is the complete rewrite that allows the new Firebrick to run using all the cores.

In reply to a post by jaydub:

We had a very short outage at 23:40 last night. It's the first time since they reverted to the stable firmware. Anyone else have any similar issues?

In reply to a post by jaydub:

We had a very short outage at 23:40 last night. It's the first time since they reverted to the stable firmware. Anyone else have any similar issues?

Edited by perlen (Thu 24-Oct-24 08:26:55)

In reply to a post by perlen:

Tuesday night: disconnections of broadband lines, 3x LNSs lost power.
Wednesday: shuffling people back around to the right LNS.
Wednesday afternoon: routing and traffic loss problems for two hours.
This morning, Thursday: massive packetloss and disruption for 45 minutes.

In reply to a post by behuk:

I'm not excusing the disruption, but I suspect a number of providers would simply not report these sort of issues. I think it's fair to say that it's quite unusual for a residential broadband provider to (e.g.) invite their customers to email support for a RFO.

We do apologise to customers affected by the problems this afternoon. Please feel free to email [email protected] for a 'Reason for Outage'

Edited by E300 (Thu 24-Oct-24 09:10:39)

In reply to a post by E300:

In reply to a post by behuk:

I'm not excusing the disruption, but I suspect a number of providers would simply not report these sort of issues. I think it's fair to say that it's quite unusual for a residential broadband provider to (e.g.) invite their customers to email support for a RFO.

Well given how transparent they say they are, they are now managing the bad PR by hiding the blip chart on their status page and not publicly saying what the issues are, but you can email in, perhaps sign an NDA before they tell you

We do apologise to customers affected by the problems this afternoon. Please feel free to email [email protected] for a 'Reason for Outage'

Blip chart is accessible still here: https://control.aa.net.uk/blip.cgi

Perhaps this is a new issue with their Firebrick 9000? Did they ever fix the random crashing issue or are they still using the last firmware that was stable?

Haven't they also replaced all their older but stable LNSs with the 9000s now?

Thankfully I'm an ex customer having already been put off by the last lot of troubles, currently on a 173 days of uptime without a single issue, that is from the day of switchover from AA to Unchained.

In reply to a post by behuk:

I'm not sure how effectively they've "hidden" the blip chart given it's still available via Google and their wiki. I'm struggling to find the Unchained blip chart though?

In reply to a post by clnsp2:

I contacted them regarding it and received an open and detailed summary, as usual.

Edited by Chrysalis (Thu 24-Oct-24 13:20:34)

In reply to a post by behuk:

I'm not sure how effectively they've "hidden" the blip chart given it's still available via Google and their wiki. I'm struggling to find the Unchained blip chart though?

Edited by E300 (Thu 24-Oct-24 13:29:07)

Edited by perlen (Thu 24-Oct-24 21:48:36)