Downloading graphs from script

In order to diagnose network problems, I'm setting up a script to download every day's BQM and archive them. If the script gets called just after midnight, and the network is up, I can use

curl 'https://www.thinkbroadband.com/broadband/monitoring/quality/<id>.png'

and get a well aligned graph, but obviously that's very dependant on timing, and the network being stable at that point. At any point today, I can use

curl 'https://www.thinkbroadband.com/broadband/monitoring/quality/<id>-18-09-2020.png'

to get a well aligned graph for yesterday, but that only seems to work sometimes - it often gets redirected to the login page. I've done a lot of fiddling with cookiejars, and copying cURL commands from Chrome Inspect, but I can't get it to work reliably in the long term.

Is there an approved way of authenticating these kind of calls, or can I make my BQM history public so it doesn't need authentication?

I've tried the "Share" button, but it only makes the live view, or a specific day's graph public, not all my history

Thanks for any tips

I think this thread might explain why you often need to log in again.

Interesting, thanks! I'd really rather avoid cookies entirely - for this reason among many others. Access tokens provide a much more reliable way for scripts to interact with websites

Then again, this isn't exactly sensitive data. I'd be happy to have it publicly available without authentication

Are you logging in at any point in time? If so, how?

I typically use wget for this sort of thing as you're able to pass along the login information. You may be able to do this in curl as well, but I'm not familiar with it.

I can log in via Chrome, obviously.

curl does support HTTP authentication, using -u username:password, but ThinkBroadband ignores that. It wants to do everything via Javascript and forms. Some of this can be done with curl, but if you get one thing wrong, it's detected as a CSRF attack and you get locked out for 30 minutes, plus it means parsing the login page HTML to get the tokens etc - all very hacky and unreliable

A proper REST API would be much easier

Well, it might take you a bit before you get it right, but I'd definitely fix your login issue in curl as that's the root cause. I dealt with this on a reboot script I made for one of my older routers and once I hammered out the login, it worked smooth. I think it will be the same for you once you get the login fixed.

I wouldn't expect an api. This is just extra work for the site owners and is a attack vector or baddies.

The trouble with getting a script to parse an HTML page designed for humans, is that any formatting or layout change is likely to break it. And if they add a CAPTCHA, I'm screwed of course.

An easier option would be the choice to make one's whole BQM history public. At the moment, you can share the live graph, or a single day's history, but not all of them. I can't see how that would be a security risk - why would a baddie care what my ping time was last week?

Well, I know I wouldn't want my info public because who knows what 'bad actors' are willing to do these days and how they are going to do it. Any hole in a system and they're looking to exploit it--stupid third world mentality which [censored] me off. I'd have no problem with my info public if the bad actors were cut off from the Internet. But since they won't be, instead I have to safeguard myself from them. It's the same thing you do when you lock yourself in your house when it's not safe outside. Yay the freedom the Internet has brought--NOT.

Data Protection may come into it. Not for the ping times, but the fact they are logged alongside your ISP.

Logging in does of course protect any other information you put in your Profile (account details), visible to other logged in users. Such as your changeable registered email address that you might not want open to the world.

It also allows you to post Comments to the News Articles, where anonymous posting could/would cause chaos.

Sorry, I think you've misunderstood. I'm not saying login should never be required. But there's already an option to make your graphs public, using the Share button here
https://www.thinkbroadband.com/my-profile#bqm

It allows you to make the live, or a specific previous day's graphs public, including data protection warnings. I'm just requesting an additional options to make all previous days' graphs public, via a structured link. My profile, comments etc would still be protected by my login

Yes, I hadn't quite understood what you wanted, but!

I could have written a lot more in my previous post, but every time I added one consideration more came into play. I could have ended up with 500-multi-1000 word essay . So I deliberately kept it simple.

Your suggestion brings into play how computer programming, database storage and retrieval, how to display the list to both logged in users and anonymous readers who may be truly anonymous or members simply not logged in, (bear in mind that many may not want to have the cookie stored and be "permanently" logged in), and a possible hacking window due to a concentration of possibly a few thousand links to the database all in a single list. Though the final one of those is a suggestion above my paygrade as to its feasibility.

Bear in mind that the list you see is within the tbb website, and specifically only visible to you. Live only for a short period.

Also you may not be aware that most of the technical forums are crawled by the search engines and no doubt other less legitimate bodies.

What would be the effect on the thinkbroadband servers if an instigator of DDoS attacks discovered your list of over a thousand, (that's less that three years at one link per day, made worse if you do more than one test a day). Several hundred malware-affected computers placing simultaneous requests for access to that list, where each entry is separated by many from other users.

DDoS attacks are not necessarily from someone with a grudge against the target site. They can be from any nutter who finds it great fun to crash a website.