Home / Works / Public Works Project Scraper

Public Works Project Scraper

Python · Data Collection
Python · Playwright · pandas · QGIS

Overview

Collects public infrastructure project records page by page, filtered by year and region, flattens the JSON and exports it to Excel; a companion QGIS model processes field inspection KMZs.

How it works

Pages are fetched in a loop and stored raw; a separate model handles inspection reports.

Year / region filter(optional) Open public site inautomated browser Request one page(3 retries, pause) Raw JSON savedone file per page More pages? Read all raw pages;flatten selected fields Excel workbook Field inspectionKMZ reports Merge, expand HTMLdescription Stamp inspection date onissues, notes, remarks Layer with geometryattributes, fields dropped yes: next page no Records collection Inspection reports (QGIS model)

InputStepDecision / optionData storeOutput