I am not an A&A customer but I have to commend you on your unexpected transparent and complete response, its what sets you apart from the masses and I hope you get to the bottom of this blip.

Thanks for the update.

One question, why update all the LNS's when you think you have fixed it? I would suggest picking the LNS that seems to have had the most lockups and just updating that single one would be the prudent thing to do, then deciding on a period of time it must run crash free before you say its fixed, and only when its proved itself update the others.The overnight drops for firmware upgrades/downgrades only adds to the perception of things being even more broken.

I will log into the test LNS later today for the weekend, but during the week due to working from home I'll switch back.

I doubt any other ISP develops its own core equipment.

I think you are finding out why they don't

Edited by E300 (Fri 12-Apr-24 17:38:49)

Are all of these dropouts the fault of A&A? Are any of them Openreach or who ever?

In reply to a post by Rhynchelma:

Are all of these dropouts the fault of A&A? Are any of them Openreach or who ever?

Some recent drops were a problem created by suppliers, they typically happen overnight.

The problem we are discussing here has been an issue since November last year, where randomly and often during the day the connection has dropped for a quantity of customers because of a hardware problem with A&A's Firebrick they design themselves.

Apart from the recent BT issue that caused a lot of drops over a single night, I don't recall any other drop that wasn't caused by A&A and this issue, or due to A&A having to upgrade or downgrade firmware in order to try and fix it. To put things into perspective, the drops were not very often, I perhaps had one every few weeks, but less so now as they've reverted to stable firmware.

Edited by E300 (Fri 12-Apr-24 17:59:48)

Obviously your situation is highly specialist and unique and so I suspect might have some bearing on the following thought, but, this is getting somewhat high profile and I am wondering why there has been no mention or consideration of getting specialist 3rd parties in to put a 2nd pair of eyes on the situation? I could imagine there would be no value in help with the software, but couldn't the hardware have an issue which a specialist could possibly identify?

I know this is potentially highly costly and often does not yield anything productive, however I know based on situations at work where I have been dealing with an outage or problem, if I couldn't make progress on something so high impacting after a certain period of time, I'd be exploring means of getting a 3rd party in even as means of simply reassuring customers and end users.

FWIW, I've not been affected whatsoever by these problems (either it's because I'm on <80Mbps or I just haven't noticed), however, I have been following it out of curiosity and the lack of mention of getting external help is something that to me has been noticably missing or at least not addressed.

In reply to a post by E300:

In reply to a post by Rhynchelma:

Are all of these dropouts the fault of A&A? Are any of them Openreach or who ever?

Some recent drops were a problem created by suppliers, they typically happen overnight.

The problem we are discussing here has been an issue since November last year, where randomly and often during the day the connection has dropped for a quantity of customers because of a hardware problem with A&A's Firebrick they design themselves.

Apart from the recent BT issue that caused a lot of drops over a single night, I don't recall any other drop that wasn't caused by A&A and this issue, or due to A&A having to upgrade or downgrade firmware in order to try and fix it. To put things into perspective, the drops were not very often, I perhaps had one every few weeks, but less so now as they've reverted to stable firmware.

Thanks for the clarification.

Thank you

Those with long memories will know that this isn’t the first time AAISP have had the “why don’t you just use Cisco like everyone else?” thing thrown at them and there do seem to be some good reasons.

Pretty limited utility to the "average" punter, though. The bandwidth accounting features are useful to AAISP with their commercial model, but not really to the customer, other ISPs simply don't charge for bandwidth. Much of the functionality from the CQM side can be achieved using 3rd party services (arguably even more useful, as it takes into account issues in the ISP's transit).And the bonding services must be dying a death now for residential, with rubbish ADSL lines rapidly being something folk don't need to contend with, though maybe more useful to businesses for failover (I think central bonding is a bad idea for that use case, though, better using distinct network providers with local failover).

I raised the question of the stability of the (then beta) LNS HW around a couple of years ago when I joined AAISP, I was a little concerned it could be an issue. As it would happen, over the period I was a customer the LNS behaved impeccably; I left because no matter the quality of service, the anxiety over the quota system and trying to juggle changing quotas to use up my allowance at minimum cost just annoyed me too much.

I'm with Unchained now, who use Cisco for their LNS; performance is great (900Mbps line rate single threads to Cloudvider iperf services in UK and often EU) - and no quotas at lower cost than the least expensive AAISP service. Small outfit, great personal service. I certainly don't miss any features from the FB9000 I'm (no longer) connected to. I think it is a tenuous benefit for many, and in hindsight my concern was probably founded in common sense (and I got lucky over my tenure).

In reply to a post by jimbof:

Pretty limited utility to the "average" punter, though. The bandwidth accounting features are useful to AAISP with their commercial model, but not really to the customer, other ISPs simply don't charge for bandwidth. Much of the functionality from the CQM side can be achieved using 3rd party services (arguably even more useful, as it takes into account issues in the ISP's transit).And the bonding services must be dying a death now for residential, with rubbish ADSL lines rapidly being something folk don't need to contend with, though maybe more useful to businesses for failover (I think central bonding is a bad idea for that use case, though, better using distinct network providers with local failover).

I raised the question of the stability of the (then beta) LNS HW around a couple of years ago when I joined AAISP, I was a little concerned it could be an issue. As it would happen, over the period I was a customer the LNS behaved impeccably; I left because no matter the quality of service, the anxiety over the quota system and trying to juggle changing quotas to use up my allowance at minimum cost just annoyed me too much.

I'm with Unchained now, who use Cisco for their LNS; performance is great (900Mbps line rate single threads to Cloudvider iperf services in UK and often EU) - and no quotas at lower cost than the least expensive AAISP service. Small outfit, great personal service. I certainly don't miss any features from the FB9000 I'm (no longer) connected to. I think it is a tenuous benefit for many, and in hindsight my concern was probably founded in common sense (and I got lucky over my tenure).

Yes there was a decent spell without issues. I agree that with FTTP there is less of a need for the bespoke monitoring they have which I've never needed. Perhaps they should move to tried and already tested third party kit for their FTTP customers as it seems a bit cheeky to use their customers as unpaid beta testers. Their unique selling point, ironically, is the reason I'm now looking at other options.

Their usage caps, well I'm borderline between the two options so just opt for the higher tier so I don't find myself checking usage towards the end of the month and rationing it. Given the issues recently I don't find myself quite as okay with paying over the odds, yet I don't want to go back to metering my connection by moving to their lower tier. Another reason to look elsewhere.

Unchained might be a contender. I did see on their website mention of line bonding and "Firebrick" was mentioned, so I did wonder if they were using Firebricks throughout. Is it definitely Cisco they use? It would be very annoying if after a few months they upgraded their Firebricks or the firmware and they get the same problems.

Edited by E300 (Sun 14-Apr-24 09:37:21)

In reply to a post by aabloor:

We do still believe strongly that the FB9000, when stable, offers us features that distinguish our service from the service of almost all others. Simply, we want bonding, CQM graphs, low power consumption, etc.

It is part of what makes our ISP offering different and better; our USP.

Hey Alex,

Appreciate the rest and it's admirable, you make great points and I hope they are accepted with grace and good manners, but a couple of questions for my own interest on the quoted section and another little bit.

Bonding: you folks are only putting customers on 300 Mbit+ onto the FB9000, indicating they are on FTTP. Is there much of a market for bonding FTTP? The only use case I can think of is a very high end residential/SME using you guys across 2 different physical FTTP networks, so needing CityFibre and Openreach FTTP available to them. Using Openreach FTTP via BT Wholesale and TalkTalk Business is a use case however the protection is substantially reduced to the point where an active-backup solution may as well be used.

For capacity aside from racking up multiple Openreach services for higher upload due to their gross asymmetry I can't see another use case. I'm thinking there won't be that many desiring such capacity using your services as, as I recall, your usage per subscriber is very much towards the lower end of the market.

Regardless bonding can be done without PPP on carrier and not so carrier kit as I'm sure you folks are aware. Commodity servers are amazing at LNS duty, FPGA even better, ASIC even better still.

CQM had enormous value during copper times. The only folks on the 9000 are on FTTP. Issues with performance across your carriers are very few and far between now. Utilisation can be measured at the BNG, latency and loss rely on preferential treatment of LCP and without it may be measured out of band with similar accuracy so what customer benefit do you see going forward?

Another question related is are you folks going to use PPP indefinitely given a major feature relies on it? Openreach deliver Ethernet, CityFibre deliver Ethernet, at some point BT Wholesale will deliver Ethernet, they wanted to move away from PPP when GEA started, TTW etc will follow. Are you folks going to be having the CPE putting customer data in PPP purely for the LCP echoes to keep CQM running?

Lower power consumption is super important but per subscriber how are the numbers?

I note according to the website the FB9000 has a pair of 10GbE ports, the only two on each appliance, that seem to share a single 10 gigabit backplane to the CPU. You folks start selling 1.8 and 2.3 Gbit services the symmetrical ones especially are going to be problematic: a customer uploading 2.3 Gbit/s is taking 2.3 Gbit/s of downstream capacity from the backplane and as we know saturating downstream for any length of time is hard, saturating upstream, which you folks do not meter, isn't. With relatively skinny pipes and relatively high burst compared to the pipes your LNS end up looking less like LNS and more like transport links with high burst to sustained ratio. You can mitigate this with racks and racks of LNS as you did the smaller models leaving fewer subscribers per chassis but where does that leave power consumption per subscriber, the metric that matters?

With the advent of DTT going IPTV the base load on networks is going to increase substantially and with higher burst products to continue to never be the bottleneck you're potentially going to have to provision a lot of kit and have relatively few subscribers on each chassis. As I mentioned above I remember your usage announcements per subscriber being at numbers many ISPs would envy as they were running at twice or more those numbers. DTT and other services going all-IP will close that gap a ton and your usage will jump reducing customers per appliance even more.

The FB9000 is brand new, can't be purchased yet, what kind of lifespan do you folks see for it?

I think there's a big asymmetry inherent in this part:

In reply to a post by aabloor:

This is where our two roles; that of both an ISP with broadband customers, and also that of a hardware manufacturer meet each other head-on and, unfortunately and uncomfortably, collide.

Without using Firebrick the ISP can still be bloody good, IMHO without CQM excellent reactive and proactive support is super important: I don't use you folks because FTTP doesn't really break often so high level support isn't really needed, I have an excellent altnet as my primary service, a personal friend provides my backup and regardless I have my own monitoring due to my job else you guys would be my backup. Without the ISP using Firebrick Firebrick is probably not so healthy putting a fair amount of pressure on the ISP to continue using Firebrick regardless and ensure the business remains viable.

On another note thank you again for my Ignis: he remains in the background of every work Zoom and Teams call as he's just the cutest <3

Edited by XGS_Is_On (Sun 14-Apr-24 12:37:26)