How It Works
From raw weather data to calibrated probability to financial risk products — the Cliff Horizon pipeline.
Cliff Horizon's engine transforms raw weather data into actionable financial risk products through a five-stage pipeline.
The Pipeline
Stage 1 — Data Ingest
The engine ingests data from three independent layers every six hours, aligned to NWP model runs (00Z, 06Z, 12Z, 18Z).
Layer 3: Meteorological (Public NWP)
- GFS — 31-member ensemble, 0.25° resolution, 16-day horizon
- ECMWF — 51-member ensemble, 0.25° resolution, 15-day horizon
- HRRR — 3km resolution over CONUS, 48-hour horizon
Accessed via the Open-Meteo Ensemble API. No API key required.
Layer 2: Behavioural
- Grid load data from EIA Hourly Electric Grid Monitor — covers PJM, ERCOT, CAISO, NYISO, ISO-NE, MISO, SPP
- Electricity demand responds directly to temperature. A load anomaly at 2pm carries forward-looking information about temperature that NWP models may not yet reflect.
Layer 1: Satellite (SatSure Sparta)
- Land surface temperature, soil moisture, vegetation indices, cloud cover
- Proprietary satellite-derived observations unavailable to the public NWP ensemble crowd
- Provides ground truth for bias correction — what the weather actually did, measured from space
Stage 2 — Bias Correction
Raw NWP forecasts contain systematic biases that vary by location, season, and weather regime. The engine applies city × season × regime-specific corrections:
- Warm bias in summer high temperatures (documented across NWS forecasts)
- Coastal effects at airport weather stations
- Urban heat island adjustments for city-centre sites vs airport observation points
- Seasonal structural breaks where model performance changes character
Bias correction is estimated from historical forecast-vs-observed comparisons and updated continuously as new verification data arrives.
Stage 3 — Ensemble Weighting
Rather than treating all ensemble members equally, the engine applies inverse-error weighting — models that have been more accurate recently receive higher weight.
This is combined across NWP sources: if GFS has been outperforming ECMWF for a particular city and lead time, GFS receives proportionally more weight in the blended probability.
The output is a probability distribution for each weather variable at each location and lead time.
Stage 4 — Calibrated Probability
The raw ensemble probability is transformed into a calibrated probability — meaning the engine's stated confidence matches observed frequencies.
What calibration means in practice: If the engine assigns 30% probability to 100 different events, roughly 30 of those events should occur. This is measured by the Brier score and visualised in a reliability diagram on the Calibration tab.
The calibration step uses isotonic regression against historical verification data, with separate calibration curves per variable, location, and lead-time bucket.
Stage 5 — Risk Products
Calibrated probabilities feed directly into three product tiers:
| Product | What the Client Receives | Engine Output Used |
|---|---|---|
| Tier 1 — Analytics | Probability dashboards, risk scores, alerts | P(event) directly |
| Tier 2 — Warranty | Analytics + cash-paying performance guarantee | P(event) → variance threshold monitoring |
| Tier 3 — Derivative | Parametric payout on trigger event | P(event) → lossProb parameter in Ensuro policy |
For Tier 3 derivatives, the engine's calibrated probability becomes the lossProb field in Ensuro's policy structure. Ensuro's capital pool applies its own risk loading (margin of conservatism, cost of capital) to determine the final premium. The engine prices the risk; Ensuro provides the capital.
Data Architecture
Storage: TimescaleDB (PostgreSQL extension) for native time-series support with standard SQL.
API: FastAPI serving REST endpoints for each dashboard tab. WebSocket for live market price streaming.
Audit trail: Every prediction stores its full provenance — timestamp, NWP sources ingested, bias correction parameters, ensemble weights, raw and calibrated probabilities. This chain of custody is required for Tier 2 warranty verification.