General Discussion
  >> Fibre Broadband


Register (or login) on our website and you will not see this ad.


  Print Thread
Standard User SCollins
(newbie) Tue 02-Jun-15 17:35:04
Print Post

Technical help understanding data in Roadworks.org


[link to this post]
 
Recently I said I'd help find information out and after someone mentioned that all his sources were online (I had wrongly assumed some people here had insider information), so I've been looking behind the scenes at some of the main websites mentioned to see how I could automate data collection to post back here for everyone.

The first place I thought to look was RoadWorks - I have a script that polls the server for information and only extracts electricity and telecoms roadworks (should I expand?) and looks for "BDUK" within the description that is shown up on the website.

To help further reduce unwanted noise I was wondering if anyone could help me understand a little more some of the phrases that I'm seeing to see if I can bucket these into a field predicting how far along the work is (e.g.: "Initial Phase", "Final Phase" or something along those lines):

- BUILD JOINT BOX & DUCT- CIVILSTo build new JUF 106 joint box and lay approx 2m of Duct 54 in Verge
- Lay approx 3m of Duct 54 in CW/FW to link existing BT Boxes to facilitate spine cabling works.Also additional new concrete joint box in Footway (JUF106) possibly in carriageway if unable to install in F/W
- 3 way Rolling lights required for BT Openreach to install sub-duct with t… (seems to be truncated - get the lights bit but not sure what a sub-duct is)
- Remedial reinstatement
- Lay approx 121m of Duct 54 in CW/FW/ to link existing BT Boxes to fa…
- 3 Way Lights required to enable De-Silting and Cabling works to take place….
- BDUK01IBK9G3PZ10 DAY POWER AND DSLAM:STREET CABINET& POWER INSTALLATION…. Am guessing this is the most obvious one to a layman like me - putting the power to the cabinet.
- To build new JMF 106 joint box and lay approx 70m of Duct 54/...
- Overlay - Lay approx 389m of Duct 54 in CW/Verge to…

Any help would be greatly appreciated or even ideas on what to pull out the data feed that comes from Roadworks.

Here's an example of what I'm pulling out via the script at the moment:
[Type] => Tel
[Lat] => 51.09139423
[Long] => 0.03199879
[StartDate] => 2015-06-03 00:00:00.0
[EndDate] => 2015-06-16 23:59:59.0
[ToolTip] => A22 Lewes Road, Forest Row, East Sussex\n03 Jun 2015 - 16 Jun 2015\nMedium impact, delays possible
[Status] => Advanced planning
[Category] => Standard
[Description] => 299429 FOREST ROW PCP10 - BDUK01IBK9G3PZ10 DAY POWER AND DSLAM:STREET CABINET& POWER INSTALLATION....
[BDUK] => Y
Standard User PaulKirby
(experienced) Thu 04-Jun-15 11:41:26
Print Post

Re: Technical help understanding data in Roadworks.org


[re: SCollins] [link to this post]
 
Well I thought I would have a go at creating an application that gets roadwork information from roadworks.org for the fun of it.

screenshot
Mine is written in C# for ease and I had it up and running in 10 or so mins after I worked out what needs to be sent and what gets sent back which took a couple of hours to work out.

It seems to work fine, the only issue I have it you can only request up to a max of 100 records and I couldn't find anyway to get the next lot.

I am not seeing any of that info you show for my requests.
All my info is returned to me as JSON, so maybe I am requesting the information differently than you are.

Mine requests for all BT work in a requested borough in just 1 request and gets returned a list of all BT work in that area, from there I can do an individual request for each of the work jobs.

What is the postcode for that work you show?

Paul
Standard User SCollins
(newbie) Mon 08-Jun-15 11:01:54
Print Post

Re: Technical help understanding data in Roadworks.org


[re: PaulKirby] [link to this post]
 
I actually did mine in PHP to make it server-side in case I ever wanted to put a web front end to it.

Sounds like we're doing the same thing - I have to do things in two stages:

1. Request "Points" and parse the JSON return (for the South of the UK I get around 26k results for a 3 month period) (use Google Map coordinates to dictate geography, at present am doing about 60% of the UK).
2. Filter out by the type of icon used (so I'm just pulling yftl01y and yfel01y)
3. Request "MarkerDetail" for each point I want more info on and parse JSON response
4. Pull out information of interest into my own array (what I outputted earlier) and add a new field BDUK as a boolean based on a REGEX of the "works_desc" returned from step 3

A couple of things I thought I could use this for was:
1. Track any changes within an area (keep a historical log in a DB of some sorts)
2. Give a prediction as to what stage people are at based on common steps that are undertaken (1. install of Cab, 2. power connection, 3. fibre ducting?)

Only issues is I don't understand some of the context of what I get back so cannot build a model that translates into some kind of process. So if anyone, in the know, is able to provide me with some guidelines as to what order / how things happen or what these nuggets of information actually mean I might be able to pull something together.

Also need to know if there is anything I need to consider outside of TEL/ELEC roadworks or should I look for something more than BDUK - I know I'm not pulling up anything on the ELEC side so guessing I need to look for something else?


Register (or login) on our website and you will not see this ad.

Standard User PaulKirby
(experienced) Mon 08-Jun-15 11:35:23
Print Post

Re: Technical help understanding data in Roadworks.org


[re: SCollins] [link to this post]
 
In reply to a post by SCollins:
I actually did mine in PHP to make it server-side in case I ever wanted to put a web front end to it.
Yeah, I was tempted of doing mine in PHP so that I can store the information in MySQL.

In reply to a post by SCollins:
Sounds like we're doing the same thing - I have to do things in two stages:

1. Request "Points" and parse the JSON return (for the South of the UK I get around 26k results for a 3 month period) (use Google Map coordinates to dictate geography, at present am doing about 60% of the UK).
2. Filter out by the type of icon used (so I'm just pulling yftl01y and yfel01y)
3. Request "MarkerDetail" for each point I want more info on and parse JSON response
4. Pull out information of interest into my own array (what I outputted earlier) and add a new field BDUK as a boolean based on a REGEX of the "works_desc" returned from step 3
How are you getting 26K results returned, all I get returned is a max of 100 records.
Also you don't need to look for the l for the type of icons thats for the Traffic Light shown in my partial list below which I had to work out for myself LOL:

'A B C D E F G'
--+-+-+-+-+-+--
"r|c|l|c|0|1|y"    "High impact, delays likely"             (Red, Digging in road, no entry)
"r|c|l|l|0|1|y"    "Traffic control (multi-way signals)"    (Red, Digging in road, traffic lights)
"y|c|w|n|0|1|y"    "Medium impact, delays possible"         (Yellow, Water Tap)
"r|c|w|c|0|1|y"    "High impact, delays likely"             (Red, Water Tap, no entry)
"r|c|w|l|0|1|y"    "High impact, delays likely"             (Red, Water Tap, traffic lights)
"y|f|r|l|0|1|y"    "Medium impact, delays possible"         (Yellow, Train, traffic lights)
"r|f|t|a|0|1|y"    "High impact, delays likely"             (Red, Phone, one lane closed)
"g|c|l|g|0|1|y"    "Low impact, delays unlikely"            (Green, Digging in road, Give and take)
"g|f|w|p|2|0|y"    "Low impact, delays unlikely"            (Green, Water Tap, Priority works)
"y|f|t|s|1|8|y"    "Medium impact, delays possible"         (Yellow, Phone, Stop/Go boards)
"g|f|g|n|0|1|y"    "Low impact, delays unlikely"            (Green, Black Flame [Gas maybe])
"y|c|e|n|0|1|y"    "Medium impact, delays possible"         (Yellow, Lightning Bolt)
--+-+-+-+-+-+--
[A, Colour of icon, Red, Yellow or Green]
r = red
y = yellow
g = green

[B, E, F, G, ???]

[C, Type of icon]
l = local authority work (road digging icon)
w = water/sewage works (water tap)
r = Network Rail works (train)
t = Telecoms work (phone)
g = Gas works (black flame)
e = Electricity works

[D, Type of symbol on icon, like no entry or traffic lights etc]
n = [chunk 1]	none
l = [chunk 2]	Traffic lights.
s = [chunk 3]	Stop/Go boards
p = [chunk 4]	Priority works
a = [chunk 5]	Lane closure (lane closed)
  = [chunk 6]	Contra-flow (not seen this icon yet to work out)
g = [chunk 7]	Give and take Lane.
* Single Icon *
c = [chunk 8]	Road closure (No Entry sign)

As you can see its not fully completed.

What I do is where I am only looking for info in my area I created a POLYGON map around the area in question and that is used to selct the area I want to get infor for.
I then get sent back all work in that area (depending on the selected promoter(s) and I then process that JSON response and add them in a listview control on my app.
My intension was to have it poll for the information ever few mins (seems they update every 10 mins, so 5 mins should be fine) and then add them to the array list for each job work using its id as the key.

In reply to a post by SCollins:
A couple of things I thought I could use this for was:
1. Track any changes within an area (keep a historical log in a DB of some sorts)
2. Give a prediction as to what stage people are at based on common steps that are undertaken (1. install of Cab, 2. power connection, 3. fibre ducting?)

Only issues is I don't understand some of the context of what I get back so cannot build a model that translates into some kind of process. So if anyone, in the know, is able to provide me with some guidelines as to what order / how things happen or what these nuggets of information actually mean I might be able to pull something together.

Also need to know if there is anything I need to consider outside of TEL/ELEC roadworks or should I look for something more than BDUK - I know I'm not pulling up anything on the ELEC side so guessing I need to look for something else?
Well I plan to do that as said above, how are you filtering?
Without knowing how you are requesting your information and what page your calling my way might not be any good for you.
I know I can select areas by authority by using lst_location_area_authority and setting that to the following: "186,189" with is Newham and Redbridge councils and use that as an area insted of using a POLYGON map.
From there you can also filter by one or more promoters reducing the list even more.

BTW can you PM whet page you are sending these requests to?

Paul
Standard User SCollins
(newbie) Mon 08-Jun-15 13:16:23
Print Post

Re: Technical help understanding data in Roadworks.org


[re: PaulKirby] [link to this post]
 
I've sent you some details on PM.

Looks like you've done a lot more investigation than me - I certainly am missing some items looking at what you've put here. I'm on a videoconference right now (if only I had a fibre link!) so can't do too much that can't be noticed.

From a filtering perspective I'm purely doing a completely wrong filter on gsymbol_id but I'll correct that soon now you've given me the info! I think I'll just go for filtering on the C column for the moment. What are you thinking of doing?

Good to know they update every 5 minutes, I wonder if they have a quota system per IP to restrict people just polling them too often.

Sounds like it might be good to pool our resources on this smile
Standard User PaulKirby
(experienced) Tue 09-Jun-15 14:40:04
Print Post

Re: Technical help understanding data in Roadworks.org


[re: SCollins] [link to this post]
 
Well the updates I see are for live traffic, accidents, etc updates, the actual work checks don't seem to happen automatically and you have to just move the map to get an update of the work info, but going by the returned data that I get, I am not really sure when they actually update it, like all the last data for almost everything is last updated 14:02 for when it was last updated on roadworks.org.

So maybe its updated on road works hourly, so I would have to check in about 40 odd mins.

I like I said via PM I am using v6 of roadworks due to it allows for the search feature which I am using, so by using the filters with the search roadworks only returns the valid info that you really want instead of the several MB or so compact JSON string where you have to search through.

*** update ***
Nope the info has just updated again saying 14:32 so it seems every 30 mins.
*** end of update ***


Paul

Edited by PaulKirby (Tue 09-Jun-15 14:42:57)

  Print Thread

Jump to