General Discussion
  >> Broadband Quality Monitor


Register (or login) on our website and you will not see this ad.


Pages in this thread: 1 | 2 | (show all)   Print Thread
Standard User Doctor_Rowsdower
(newbie) Sat 19-Sep-20 11:00:55
Print Post

Downloading graphs from script


[link to this post]
 
In order to diagnose network problems, I'm setting up a script to download every day's BQM and archive them. If the script gets called just after midnight, and the network is up, I can use

curl 'https://www.thinkbroadband.com/broadband/monitoring/quality/<id>.png'

and get a well aligned graph, but obviously that's very dependant on timing, and the network being stable at that point. At any point today, I can use

curl 'https://www.thinkbroadband.com/broadband/monitoring/quality/<id>-18-09-2020.png'

to get a well aligned graph for yesterday, but that only seems to work sometimes - it often gets redirected to the login page. I've done a lot of fiddling with cookiejars, and copying cURL commands from Chrome Inspect, but I can't get it to work reliably in the long term.

Is there an approved way of authenticating these kind of calls, or can I make my BQM history public so it doesn't need authentication?

I've tried the "Share" button, but it only makes the live view, or a specific day's graph public, not all my history

Thanks for any tips
Standard User RobertoS
(elder) Sat 19-Sep-20 11:23:16
Print Post

Re: Downloading graphs from script


[re: Doctor_Rowsdower] [link to this post]
 
I think this thread might explain why you often need to log in again. smile

__________________________________________________________
Sovereignty Means Sovereignty

My broadband basic info/help site - www.robertos.me.uk. Domains, sites and mail hosting - Tsohost & Ionos.
Connections: OnePlus 8 Pro max 165Mbps down, 24Mbps up on Three, and B311 4G, tbb tests normally 35-45Mpbs down, 65Mbps off-peak, 9-24 up.
========================
To argue with mindless bigots is foolish.
Standard User Doctor_Rowsdower
(newbie) Sat 19-Sep-20 11:31:29
Print Post

Re: Downloading graphs from script


[re: RobertoS] [link to this post]
 
Interesting, thanks! I'd really rather avoid cookies entirely - for this reason among many others. Access tokens provide a much more reliable way for scripts to interact with websites

Then again, this isn't exactly sensitive data. I'd be happy to have it publicly available without authentication


Register (or login) on our website and you will not see this ad.

Standard User SamirD
(learned) Sun 20-Sep-20 14:38:14
Print Post

Re: Downloading graphs from script


[re: Doctor_Rowsdower] [link to this post]
 
Are you logging in at any point in time? If so, how?

I typically use wget for this sort of thing as you're able to pass along the login information. You may be able to do this in curl as well, but I'm not familiar with it.
Standard User Doctor_Rowsdower
(newbie) Sun 20-Sep-20 15:09:30
Print Post

Re: Downloading graphs from script


[re: SamirD] [link to this post]
 
I can log in via Chrome, obviously.

curl does support HTTP authentication, using -u username:password, but ThinkBroadband ignores that. It wants to do everything via Javascript and forms. Some of this can be done with curl, but if you get one thing wrong, it's detected as a CSRF attack and you get locked out for 30 minutes, plus it means parsing the login page HTML to get the tokens etc - all very hacky and unreliable

A proper REST API would be much easier
Standard User SamirD
(learned) Sun 20-Sep-20 15:27:05
Print Post

Re: Downloading graphs from script


[re: Doctor_Rowsdower] [link to this post]
 
Well, it might take you a bit before you get it right, but I'd definitely fix your login issue in curl as that's the root cause. I dealt with this on a reboot script I made for one of my older routers and once I hammered out the login, it worked smooth. smile I think it will be the same for you once you get the login fixed.

I wouldn't expect an api. This is just extra work for the site owners and is a attack vector or baddies.
Standard User Doctor_Rowsdower
(newbie) Sun 20-Sep-20 17:09:53
Print Post

Re: Downloading graphs from script


[re: SamirD] [link to this post]
 
The trouble with getting a script to parse an HTML page designed for humans, is that any formatting or layout change is likely to break it. And if they add a CAPTCHA, I'm screwed of course.

An easier option would be the choice to make one's whole BQM history public. At the moment, you can share the live graph, or a single day's history, but not all of them. I can't see how that would be a security risk - why would a baddie care what my ping time was last week?
Standard User SamirD
(learned) Sun 20-Sep-20 17:29:50
Print Post

Re: Downloading graphs from script


[re: Doctor_Rowsdower] [link to this post]
 
Well, I know I wouldn't want my info public because who knows what 'bad actors' are willing to do these days and how they are going to do it. Any hole in a system and they're looking to exploit it--stupid third world mentality which [censored] me off. I'd have no problem with my info public if the bad actors were cut off from the Internet. But since they won't be, instead I have to safeguard myself from them. It's the same thing you do when you lock yourself in your house when it's not safe outside. Yay the freedom the Internet has brought--NOT.
Standard User RobertoS
(elder) Sun 20-Sep-20 17:32:05
Print Post

Re: Downloading graphs from script


[re: Doctor_Rowsdower] [link to this post]
 
Data Protection may come into it. Not for the ping times, but the fact they are logged alongside your ISP.

Logging in does of course protect any other information you put in your Profile (account details), visible to other logged in users. Such as your changeable registered email address that you might not want open to the world.

It also allows you to post Comments to the News Articles, where anonymous posting could/would cause chaos.

__________________________________________________________
Sovereignty Means Sovereignty

My broadband basic info/help site - www.robertos.me.uk. Domains, sites and mail hosting - Tsohost & Ionos.
Connections: OnePlus 8 Pro max 165Mbps down, 24Mbps up on Three, and B311 4G, tbb tests normally 35-45Mpbs down, 65Mbps off-peak, 9-24 up.
========================
To argue with mindless bigots is foolish.
Standard User Doctor_Rowsdower
(newbie) Sun 20-Sep-20 21:35:34
Print Post

Re: Downloading graphs from script


[re: RobertoS] [link to this post]
 
Sorry, I think you've misunderstood. I'm not saying login should never be required. But there's already an option to make your graphs public, using the Share button here
https://www.thinkbroadband.com/my-profile#bqm

It allows you to make the live, or a specific previous day's graphs public, including data protection warnings. I'm just requesting an additional options to make all previous days' graphs public, via a structured link. My profile, comments etc would still be protected by my login
Pages in this thread: 1 | 2 | (show all)   Print Thread

Jump to