bcgov
Cloud API for Weather Data Sharing & Integrations
Target Group: Industry Researchers, Partners, and Interested Parties who want to consume BCWS Weather Data
Focus: Our focus is making BCWS Weather Data publicly available through a public API that is accessible, interactive, and offers a programmable method for querying and researching data; We believe this will deliver an improved and modern method for consuming and monitoring this data source; This will be confirmed when: a) We can shut down CSV files on the datamart ftp site b) We receive positive feedback from our end users c) The BCWS Predictive Services inbox no longer receives technical support questions from the public and stakeholders regarding how to access weather data.
Proposed Solution https://developer.gov.bc.ca/docs/default/component/aps-infra-platform-docs/
https://developer.gov.bc.ca/docs/default/component/aps-infra-platform-docs/tutorials/quick-start/
https://chat.developer.gov.bc.ca/channel/aps-ops
Product Goals in order of Priority:
- No performance issues w/ operational database when users are interacting with API. Some users may be accessing hourly weather data for a subset of stations every hour. Other users may wish to access the entire historical database for all stations in the province.
- System notifies file in ftp or similar when huge data packages are ready after a complicated/huge query.
- User documentation is available.
- Advanced search/query parameters: Search by station id or station name; by date range; within a defined geographic extent; for values above OR below a weather/FWI threshold.
- Query station data from an interactive map.
Key Performance Indicators KPI1 = Eliminate Throttling (current limitation is 365 records at any time otherwise the system will shut down access to ALL users) KPI2 = Strong UX & Documentation to eliminate support calls/emails KPI3 = PO to provide current cost/effort via Vendor: cost benefits guidance on current solution development + maintenance cost, cost to make updates, troubleshooting costs
Cadillac Version (out of scope) Landing page for subscription updates/change notifications
Additional Context
- The data that need to be exposed are stored in a Oracle Database on an NRIDS server and managed by Vivid. The main Oracle server alias is db-wfoprd1.nrs.bcgov. The main server is mirrored to a secondary alias (db-wfoprd2.nrs.bcgov) when higher load is placed on the main server. Read-only traffic should be using the secondary alias.
- A copy of these data are also stored on a public FTP site as CSV files here: https://www.for.gov.bc.ca/ftp/HPR/external/!publish/BCWS_DATA_MART/
- In the future, a cloud database will need to automatically mirror the operational Oracle Database.
- This proof of concept could use CSV files from the public FTP site to create a static cloud database and an associated API. Mirroring the operational database could happen at a later time.
Current solution in place that needs revamp - please note that no userid + pass is required anymore: BCWS Data Mart Specification.pdf
It looks like this is the current CSV server: https://www.for.gov.bc.ca/ftp/HPR/external/!publish/BCWS_DATA_MART/
https://ftsinc.com/product/autocaller-autocaller/
https://aem.eco/product/elements-360/
https://developer.gov.bc.ca/docs/default/component/public-cloud-techdocs/get-started/bc-govs-aws-landing-zone-overview/