Skip to content

Commit 61c241a

Browse files
Sam-BollingSam-Bolling
committed
Record publisher expansion checkpoint
1 parent 96119de commit 61c241a

5 files changed

Lines changed: 241 additions & 0 deletions

File tree

‎docs/research/new-publisher-source-planning/Met_Office_DataHub_Publisher_Implementation_Plan_2026-05-26.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Met Office DataHub Publisher Implementation Plan
2+
3+
Date: 2026-05-26
4+
5+
## Objective
6+
7+
Start the fourth new publisher from the original candidate set using Met Office Weather DataHub Land Observations.
8+
9+
This is an access-gated implementation: the publisher should not claim live readiness until a valid Weather DataHub subscription credential is available and tested.
10+
11+
## Source Summary
12+
13+
Met Office Weather DataHub Land Observations provides recent hourly weather observations from ground-based UK observing stations.
14+
15+
Research findings:
16+
17+
- Product: Land Observations
18+
- Domain: roughly 150 UK station locations
19+
- Format: JSON
20+
- Timesteps: hourly observations for the past 48 hours
21+
- Parameters: 9 documented parameters
22+
- Update frequency: hourly
23+
- Access: account/subscription/API credential required
24+
- Free plan: up to 360 calls per day
25+
26+
Documented API flow:
27+
28+
1. Call `GET /observation-land/1/nearest` with latitude/longitude or geohash.
29+
2. Cache the nearest land observation location/geohash.
30+
3. Call `GET /observation-land/1/{geohash}` for observations from that location.
31+
32+
## Recommended Model
33+
34+
Use `publishers/usgs_water` as the primary station-network exemplar.
35+
36+
CSAPI resources:
37+
38+
- Procedure: `urn:os4csapi:procedure:met-office-datahub-land-observations:v1`
39+
- Systems: one per curated Met Office land observation location
40+
- Datastreams: one per selected meteorological parameter per location
41+
- Deployment: one root demo deployment with child station deployments
42+
43+
Observation result fields should include:
44+
45+
- source phenomenon/result time
46+
- source geohash or location identifier
47+
- parameter name/code
48+
- value
49+
- unit
50+
- source response metadata allowed by terms
51+
52+
## Curated Demo Set
53+
54+
Choose three locations after credentials are available and nearest-location lookup can be validated:
55+
56+
1. Coastal/weather-impact station
57+
2. Urban or airport-adjacent station
58+
3. Upland/rural reference station
59+
60+
Keep polling comfortably below the free-plan limit. For a three-station demo, hourly polling with one cached nearest lookup per station and one observations call per station per hour should be safe.
61+
62+
## Implementation Steps
63+
64+
1. Verify credentials and exact request shape.
65+
2. Probe nearest-location lookup for candidate demo coordinates.
66+
3. Save resolved geohash/location metadata in a sidecar file.
67+
4. Implement `bootstrap_met_office_datahub.py` with minimal GeoJSON create stubs and rich SensorML updates.
68+
5. Implement `met_office_datahub_publisher.py` with `--dry-run`, `--once`, and interval controls.
69+
6. Add source docs, pricing/access notes, and terms links to SensorML documents.
70+
7. Run dry-run probes, then bootstrap, then one live publish cycle.
71+
8. Add Explorer role/symbol/thumbnail polish only after confirming source metadata and licensing for any representative image.
72+
73+
## Open Questions
74+
75+
- Exact API key header/query parameter for the subscribed product.
76+
- Exact JSON field names for station metadata and parameter observations in live responses.
77+
- Reuse/licensing terms for display thumbnails or Met Office brand imagery.
78+
- Whether source terms allow storing selected response metadata in OSH/CSAPI for demo purposes.
79+
80+
## Current Repository Start
81+
82+
Created:
83+
84+
- `publishers/met_office_datahub/README.md`
85+
- `publishers/met_office_datahub/__init__.py`
86+
87+
Updated:
88+
89+
- `publishers/README.md`
90+
- `Publisher_Expansion_Status_Report_2026-05-26.md`

‎docs/research/new-publisher-source-planning/Publisher_Expansion_Status_Report_2026-05-26.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Publisher Expansion Status Report
2+
3+
Date: 2026-05-26
4+
5+
## Summary
6+
7+
The first three new publisher sources are implemented, documented, pushed, and live-smoke tested against the OS4CSAPI OSH endpoint:
8+
9+
- Environment Agency Hydrology
10+
- UK-AIR
11+
- BGS SensorThings / UKGEOS Glasgow telemetry
12+
13+
The fourth candidate, Met Office Weather DataHub Land Observations, has been started as an access-gated implementation target. The source is valuable, but requires account/subscription credentials before a live publisher can be completed honestly.
14+
15+
## GitHub Issue Filed
16+
17+
Created OS4CSAPI-controlled issue:
18+
19+
- OS4CSAPI/OSHConnect-Python#7: `System SensorML PUT returns HTTP 500 for publisher system metadata updates`
20+
21+
The issue tracks the repeated live OSH `PUT /systems/{id}` SensorML HTTP 500 behavior seen during rich system metadata updates. This affects normal propagation of image/source/legal metadata, even though systems, datastreams, deployments, and observations publish successfully.
22+
23+
## Production Smoke Sweep
24+
25+
Production Explorer assets were checked after the BGS thumbnail deploy:
26+
27+
- Production JS: `/assets/index-C5SpiU1M.js`
28+
- Production CSS: `/assets/index-D9hdi0bV.css`
29+
- CSS response: HTTP 200, `text/css`
30+
- BGS thumbnail fallback URL present in production JS
31+
32+
Production CSAPI deployment read-back confirmed these publisher groups:
33+
34+
| Publisher | Deployment ID | UID |
35+
| --- | --- | --- |
36+
| Environment Agency Hydrology | `05d0` | `urn:os4csapi:deployment:environment-agency-hydrology-demo:v1` |
37+
| UK-AIR | `05g0` | `urn:os4csapi:deployment:uk-air-demo:v1` |
38+
| BGS SensorThings | `05ig` | `urn:os4csapi:deployment:bgs-sensorthings-demo:v1` |
39+
40+
Live publisher smoke results:
41+
42+
| Publisher | Result |
43+
| --- | --- |
44+
| Environment Agency Hydrology | Connected 4/4 stations, published 5 observations, 0 errors |
45+
| UK-AIR | Connected 3/3 stations, published 4 observations, 0 errors |
46+
| BGS SensorThings | Source dry-run fetched 9/9 curated observations, 0 errors; live run connected 3/3 stations and began posting the same curated readings |
47+
48+
Browser production card verification:
49+
50+
- BGS `BGS GGA07 UKGEOS Deployment` card rendered the official UKGEOS borehole-dimensions SVG thumbnail.
51+
- The image loaded with natural size `320x237`.
52+
- BGS card retained `Groundwater Telemetry Site`, latest readings, source links, and legal/attribution links.
53+
54+
Map-click automation for EA and UK-AIR became unreliable in the shared narrow browser viewport, so the smoke record uses production deployment read-back plus live publisher publish-cycle validation for those two. Earlier completion reports already record their production card verification.
55+
56+
## Reusable Implementation Pattern
57+
58+
The new publisher standard is now clear:
59+
60+
1. Research source access, licensing, and API shape before implementation.
61+
2. Curate a small demo-safe station/source set in a sidecar file.
62+
3. Bootstrap minimal GeoJSON stubs first, then rich SensorML PUTs.
63+
4. Use stable UIDs and idempotent bootstrap helpers.
64+
5. Publish one observation per source-native latest reading.
65+
6. Preserve source IDs, source URLs, observed property, unit, and timestamp in result metadata.
66+
7. Add Explorer role/symbol/label/thumbnail polish when the source introduces a new domain category.
67+
8. Document residual server/source quirks immediately in the completion report or a GitHub issue.
68+
69+
## Known Residuals
70+
71+
- Live OSH system SensorML PUT still returns HTTP 500 for affected station-system updates. Tracked in OS4CSAPI/OSHConnect-Python#7.
72+
- Explorer representative thumbnail fallbacks remain necessary for demo-quality cards when live system SensorML metadata cannot be updated.
73+
- Met Office Weather DataHub cannot be completed without subscription credentials, even though the free plan allows limited daily call volume.
74+
75+
## Publisher #4 Start
76+
77+
Met Office Weather DataHub Land Observations is the fourth source from the original candidate set.
78+
79+
Research confirmed:
80+
81+
- Land Observations covers roughly 150 UK station locations.
82+
- The API provides hourly JSON observations for the past 48 hours.
83+
- The product currently advertises 9 parameters.
84+
- The documented API flow is nearest-location lookup, then observation retrieval by geohash.
85+
- Nearest-location lookup results should be cached.
86+
- A free plan exists up to 360 calls per day, but registration/subscription/API credentials are required.
87+
88+
Created starter package:
89+
90+
- `publishers/met_office_datahub/README.md`
91+
- `publishers/met_office_datahub/__init__.py`
92+
93+
Recommended next implementation step after credentials are available:
94+
95+
1. Verify the exact subscribed API base URL and authentication header.
96+
2. Probe `GET /observation-land/1/nearest` for 3 curated demo locations.
97+
3. Cache geohash/location metadata in a sidecar file.
98+
4. Create bootstrap and runtime publisher using the `usgs_water` station-network model.

‎publishers/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ server (e.g. [OpenSensorHub](https://opensensorhub.org/)).
2121
| **Environment Agency Hydrology** | EA river level, flow, rainfall, groundwater | 15 min |
2222
| **UK-AIR** | Defra UK-AIR NO2, O3, PM10, PM2.5 | 1 h |
2323
| **BGS SensorThings** | BGS UKGEOS Glasgow groundwater/geothermal telemetry | 15 min |
24+
| **Met Office DataHub** | Land Observations hourly weather data | Access-gated / planned |
2425

2526
## Quick Start
2627

@@ -59,6 +60,7 @@ python -m publishers.usgs_eq.bootstrap_usgs_eq
5960
python -m publishers.environment_agency_hydrology.bootstrap_environment_agency_hydrology
6061
python -m publishers.uk_air.bootstrap_uk_air
6162
python -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings
63+
# Met Office DataHub is access-gated; see publishers/met_office_datahub/README.md
6264
python -m publishers.iss.bootstrap_iss
6365
```
6466

@@ -114,6 +116,7 @@ python -m publishers.nws.nws_publisher --interval 3600
114116
| `BOOTSTRAP_URL` | No | Override the full bootstrap API URL |
115117
| `OSH_FORCE_IP` | No | Force DNS resolution to a specific IP |
116118
| `USGS_API_KEY` | No | USGS API key for higher rate limits |
119+
| `MET_OFFICE_DATAHUB_API_KEY` | Planned | Met Office Weather DataHub subscription key for Land Observations |
117120
| `BUOYCAM_CACHE_ROOT` | No | Local directory for BuoyCAM image cache |
118121
| `BUOYCAM_CACHE_BASE_URL` | No | Public URL serving the cached images |
119122

@@ -130,4 +133,7 @@ python -m publishers.nws.nws_publisher --interval 3600
130133
only the curated pollutant timeseries in `stations.json`.
131134
- **BGS SensorThings** uses the public BGS Sensor Data Service SensorThings API
132135
and polls only curated unrestricted UKGEOS Glasgow datastreams in `stations.json`.
136+
- **Met Office DataHub** is started as an access-gated publisher target for Land
137+
Observations. It requires account/subscription credentials before live runtime
138+
implementation can be completed.
133139
- All publishers use `--interval <seconds>` and `--dry-run` CLI flags.

‎publishers/met_office_datahub/README.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Met Office Weather DataHub Publisher
2+
3+
Status: started, access-gated implementation target.
4+
5+
This publisher slot is reserved for the fourth candidate source from the new-publisher expansion research: Met Office Weather DataHub Land Observations.
6+
7+
## Source
8+
9+
- Weather DataHub landing page: https://datahub.metoffice.gov.uk/
10+
- Observations overview: https://datahub.metoffice.gov.uk/docs/g/category/observations/overview
11+
- Land Observations API documentation: https://datahub.metoffice.gov.uk/docs/g/category/observations/type/land-observations/api-documentation
12+
- Observations pricing: https://datahub.metoffice.gov.uk/pricing/observations
13+
14+
## Current Research Notes
15+
16+
Met Office Land Observations provides recent historical weather observations from ground-based instruments across roughly 150 UK station locations. The public documentation describes hourly JSON observations for the past 48 hours, with 9 parameters and an hourly update cadence.
17+
18+
The API flow is documented as:
19+
20+
1. Call `GET /observation-land/1/nearest` with latitude/longitude or geohash to identify the nearest land observation location.
21+
2. Cache the nearest-location result.
22+
3. Call `GET /observation-land/1/{geohash}` to retrieve observations for that land location.
23+
24+
The service is account/subscription gated. The public pricing page lists a free Land Observations plan up to 360 calls per day, but access still requires registration, product selection, subscription, and API credentials.
25+
26+
## Proposed CSAPI Model
27+
28+
Use the station-network pattern from `publishers/usgs_water`, with compatibility guardrails from `publishers/aviation_wx`.
29+
30+
- One procedure for Met Office Land Observations ingestion.
31+
- One system per curated land observation location.
32+
- One datastream per selected meteorological parameter per station.
33+
- One deployment group for the curated Met Office demo set.
34+
- Observations should preserve the source time, geohash/location identifier, observed parameter, value, unit, and source response metadata where permitted.
35+
36+
## Implementation Gate
37+
38+
Do not implement a live runtime until a `MET_OFFICE_DATAHUB_API_KEY` or equivalent subscription credential is available and the exact API request headers/query parameters are verified against the subscribed product.
39+
40+
When credentials are available, start with a small curated set of UK locations that complement existing demo publishers, for example:
41+
42+
- one coastal/weather-impact location,
43+
- one urban or airport-adjacent location,
44+
- one upland/rural reference location.
45+
46+
Keep total call volume comfortably below the free-plan limit during demo mode.

‎publishers/met_office_datahub/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Met Office Weather DataHub publisher package."""

0 commit comments

Comments
 (0)