robbrad
New year, new display method but still no official API - Gedling
Name of Council
Gedling Borough Council
Issue Information
The last of the previous iterations of the Gedling calendar has just ended and as a result they've implemented a new system to provide the information.
Sadly, they've not provided an API so the only way to find it is to search for individual addresses on their bin collection page and then view either a calendar or list table, which whilst they claim it's easier to read and more accessible they're mistaken. There is a PDF calendar that can be downloaded for these results but it's equally as terrible.
Alas, this could be the end for Gedling's data to be usable with this integration.
Verification
- [x] I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
- [x] I have checked my address/postcode/UPRN works on the council's website
- [x] I have provided a detailed explanation of the issue as well as steps to replicate the issue
- [x] I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed
My initial thoughts and findings in https://github.com/jamesmacwhite/gedling-borough-council-bin-calendars/issues/6#issuecomment-3593041175
I could still use the same process to keep the API endpoints this project is using working. The front-end search on gbcbincaledars.co.uk to map address to schedule is going to be broken now, but it is the API endpoints for the JSON this project relies on primarily.
The whole schedule concept of Monday - Friday G1-G4 would still exist, it's just now difficult to get a sample of addresses for each schedule with single address searches. Although, with a bit of effort, it would be possible to find an address that maps to each one for reference.
I'd leave it up to the project maintainers here if they want to go to potentially attempt directly using the gedling.gov.uk with the changes. My project was technically middleware to basically create APIs where no possible programmatic option exists, because of horrible PDFs. The new setup does at least have some avenues for scraping/querying data directly.
There is JSON data hiding on per address queries, which produces similar data to the unofficial JSON APIs, that might be the best bet.
My initial priority will be to try and get an address for each schedule so I can basically have 2025/26 ical data which the JSON is automatically generated from. What makes it difficult, is the schedule name is now no longer displayed on anything, so mapping this to addresses will be the hard part.
Update, while the old search https://apps.gedling.gov.uk/refuse/search.aspx is no longer providing new data, it does at least map the schedule names to queries under the 2024/25 links, which I can use to build a sample of addresses to map with so I can then generate new calendars from the new data.
Just hope Gedling doesn't pull it soon.
I have published new refuse calendars for 2025/26 which will update the JSON endpoints. As the Gedling data model has changed I can't be entirely sure the 2024/25 dataset of the mapping of the collection schedule to address is accurate for addresses in 2025/26, there's definitely some anomalies because inputting results from the legacy search into the new one led to "no collection data, contact Gedling" but it is the best I can do currently, given how the current JSON data is used.
@robbrad @dp247 I'd appreciate your input on if the UKBinCollectionData project would look to try and use the new Gedling search tool, there might be opportunities, given it is web based now, there is HTML structure and possible JSON data to scrape and use, better than the PDFs.
My intentions currently are to keep the JSON APIs active for this project and possible others for the time being without breaking existing integrations, but long term they probably need reviewing, given the whole search data changes.
My findings from my initial investigation for info:
https://gedling-waste-uat.oncreate.app/w/webpage/bin-collections seems to have multiple widgets behind it
- https://gedling-waste-uat.oncreate.app/w/webpage/635GBSHB1 - Summary of collections ordered by collection date ASC
- https://gedling-waste-uat.oncreate.app/w/webpage/620GBSHB1 - Another widget which provides the full data over a year with a calendar or list view
URL parameters for these widgets appears to be:
- webpage_hash
- webpage_token
- auth
- id
On the 620GBSHB1 widget (full data) there is a .page_widget_wrapper which has a data-params attribute which seems to contain the JSON data behind the HTML data being displayed.
If it is possible to make programmatic requests via POST with address searches, you could scrape this to basically use as a crude JSON API, even though it isn't directly designed for it, but the data is there. The issue is possibly going to be CSRF and these tokens/auth params which might be tricky from a non-interactive perspective.
The new address search has data gaps, a family member's address in Gedling doesn't show up at all, so I think there's still some data anomalies, this does seem to confirm similar issues I experienced when using the old legacy search and finding data in the legacy search appeared to not exist in the new one.
Garden collections appear to be mostly missing on all addresses, although a couple of samples seemed to show garden collections in the summary widget, while others didn't and there's no garden collection information on the full list info. The Garden collection data is still good until March 2026, so doesn't really impact this project currently.
The legacy search is still available at: https://apps.gedling.gov.uk/refuse/search.aspx
Weirdly the download calendar links have all been updated to 2026 paths but they all 404. I would find it odd they'd produce PDFs since this new search app. I suspect that perhaps the legacy search system has a rollover after 30th November and it just assumes the paths exist.
I'm going to suggest Gedling are still tweaking and developing this, given some of the anomalies found.
Hi @sym0nd0,
Good news! This has been tested and is already working ✅
Despite the council implementing a new calendar system for 2025, the Gedling Borough Council scraper is still functioning correctly and successfully retrieving bin collection data.
Next Steps:
- Update to the latest version of the integration
- Test with your address/postcode
- If it's working correctly, please close this issue
The scraper appears to have adapted to the new system or the council may have maintained API compatibility.
Thanks for reporting this!
The existing integration works, because the API endpoints on gbcbincalendars.co.uk were updated under the previous data structure of schedules being Monday - Friday, G1, G2, G3, G4. The new search tool is designed around individual addresses rather than this model now.
Old data: https://apps.gedling.gov.uk/refuse/search.aspx New data: https://waste.digital.gedling.gov.uk/w/webpage/bin-collections
The old search tool is still available, which is why my unofficial search tool and API is still able to provide street address to schedule mapping, but it appears this is now legacy or at least likely to be removed in the future given Gedling's development work on the new tool.
I was able to get sample street addresses for each schedule to check against under the old and new data and to the best of my knowledge should remain accurate, but I now can't be sure given the schedule data model is gone.
Going forward, this project may want to look at using the new search tool directly, if it possible to send POST requests programmatically without being blocked or prevented via CSRF, given Gedling are now providing the data in HTML. There appears to be JSON available too which can be used to return the data similarly to my original JSON approach, but would rely on scraping from HTML DOM source.
I am considering if I can leverage the search data from the new tool and generate ical/JSON data from this, which makes it less static, given I manually create ical data and build JSON off this, given the iCal data is the foundation data source to be usable generically without specific home automation software in mind.
I'll continue to keep the middleware APIs this project uses active, but it is going to be under the schedule name model not address in the immediate future.
I did a bit of reverse engineering behind this new dataset and the new web forms. The issue is it takes multiple POST requests to multiple endpoints to get to the full calendar data, which without interaction is going to be quite tricky with how the forms/POST requests work. I put them through some TLS/inspection tools and there's a lot of parameters and tokens which are difficult to replicate without user interaction or generate dynamically.
@sym0nd0 Have you considered reaching out to [email protected] about this new data and there are plans to provide APIs?
The system behind it appears to be: https://gedling-central.oncreate.app/ which is known as Liberty Create https://www.netcall.com/platform/liberty-create/
The tricky part is each schedule is now behind an address query. Internally I assume the schedule data i.e. Monday G1, G2, G3 etc still exists but is no longer directly visible. I can't imagine internally the schedule data is that different as the data relationship would be one schedule to many street names.
My position at the moment is as long as I can map sample addresses to the older schedule dataset, I can still maintain the calendar endpoints under that structure, the issue is this could be diverging from the latest dataset, so the accuracy cannot be 100% known.
@robbrad @sym0nd0 I have found a potential unofficial API method which uses the new bin collection search data, but gets it directly from gedling.gov.uk, I looked at the possibilities and have found trying to make programmatic requests to the new front end is too difficult and fragile given you need CSRF, Cookies, webpage tokens and such. Instead, the better approach is using a real browser (avoids all the CSRF/token issues) and performing interaction and DOM scraping to retrieve the data required.
I wrote a new API endpoint under the gbcbincalendars.co.uk project as a proof of concept/test
http://api.gbcbincalendars.co.uk/get-bin-collection-calendar?address=101 Arnold Lane Gedling Nottingham NG4 4HE
Using the sample address 101 Arnold Lane Gedling Nottingham NG4 4HE, this endpoint will automatically:
- Navigate to https://waste.digital.gedling.gov.uk/w/webpage/bin-collections
- Trigger the autocomplete search from the search input and select the first result
- Go to view collection days
- View the full 2025/26 collection data for the returned address
- Locate the widget data-params which provides full data as JSON
- Extract the JSON and return the collection data information as a JSON response.
Example JSON:
[
{
"month": "2025-12",
"dates": [
{
"date": "2025-12-05",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2025-12-12",
"collections": [
"General Collection Service"
]
},
{
"date": "2025-12-19",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2025-12-27",
"collections": [
"General Collection Service"
]
}
]
},
{
"month": "2026-01",
"dates": [
{
"date": "2026-01-03",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-01-09",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-01-16",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-01-23",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-01-30",
"collections": [
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-02",
"dates": [
{
"date": "2026-02-06",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-02-13",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-02-20",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-02-27",
"collections": [
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-03",
"dates": [
{
"date": "2026-03-06",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-03-13",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-03-20",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-03-27",
"collections": [
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-04",
"dates": [
{
"date": "2026-04-03",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-04-10",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-04-17",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-04-24",
"collections": [
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-05",
"dates": [
{
"date": "2026-05-01",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-05-08",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-05-15",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-05-22",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-05-29",
"collections": [
"General Collection Service"
]
}
]
},
{
"month": "2026-06",
"dates": [
{
"date": "2026-06-05",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-06-12",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-06-19",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-06-26",
"collections": [
"General Collection Service"
]
}
]
},
{
"month": "2026-07",
"dates": [
{
"date": "2026-07-03",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-07-10",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-07-17",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-07-24",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-07-31",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-08",
"dates": [
{
"date": "2026-08-07",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-08-14",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-08-21",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-08-28",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-09",
"dates": [
{
"date": "2026-09-04",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-09-11",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-09-18",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-09-25",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
}
]
},
{
"month": "2026-10",
"dates": [
{
"date": "2026-10-02",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-10-09",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-10-16",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-10-23",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-10-30",
"collections": [
"General Collection Service"
]
}
]
},
{
"month": "2026-11",
"dates": [
{
"date": "2026-11-06",
"collections": [
"Recycling Collection Service"
]
},
{
"date": "2026-11-13",
"collections": [
"General Collection Service"
]
},
{
"date": "2026-11-20",
"collections": [
"Glass Collection Service",
"Recycling Collection Service"
]
},
{
"date": "2026-11-27",
"collections": [
"General Collection Service"
]
}
]
}
]
As a proof of concept, it works, but there are a few issues. The address search is the somewhat annoying part. Behind the scenes, there appears to be some form of ID for each property, for example 101 Arnold Lane Gedling Nottingham NG4 4HE is 65461. This looks like some internal ID, rather than a general UPRN. The address autocomplete is ironically the least accessible part of this new system and a bit of a faff to make work with automated browser testing, but by using full address searches in the Royal Mail PAF format, you can automatically send address queries where the first returned result is assumed to be correct, basically exact searches.
From there it is basically chaining multiple interactions (clicks) and then scraping JSON data from the web page to provide the collection which can be consumed by other applications.
From my reverse engineering efforts, the system behind this clearly has data sources available, so this is possibly over engineered when there are likely much easier API layers available, but whether these are public or Gedling intends to make them is another story.
For example, hiding in the Content Security Policy of their website is some UAT API endpoint: https://gbcapi-municipal-uat.whitespacews.com/, this is a SOAP web service https://gbcapi-municipal-uat.whitespacews.com/WSAPIService.svc?wsdl which has a lot of endpoints. Several interesting ones e.g. GetCollectionByUprnAndDatePlus or GetAddresses, which would suggest there's an API layer behind it all. I wasn't able to query anything however, likely security/authentication required.
From my own adventures, if this project is ever looking to use this data directly, you are likely going to need to use selenium. I used puppeteer, because of the existing API endpoint being a serverless Cloudflare Worker, but the same concept would apply. You'd basically need to use a real browser to avoid the CSRF/token issues and then scrape the results.
Of course, if Gedling drop official APIs, then this becomes a lot easier, but councils often tend to not do this, hence the existence of this project to begin with!
Closing as its covered in #1839. Huge thanks to @jamesmacwhite fantastic work