Documentation
Technical reference for Grid Poet
1. DA Price Forecast
Grid Poet forecasts day-ahead electricity prices for the DE-LU bidding zone (Germany + Luxembourg). The forecast is generated daily at 07:00 CET — well before the EPEX SPOT order book close at 12:00 CET — and predicts 24 hourly prices for the next delivery day (D+1).
The model outputs three quantiles: P10 (10th percentile, lower bound), P50 (median, point forecast), and P90 (90th percentile, upper bound), forming an 80% prediction interval.
1.1 Pipeline Flow
The daily forecast pipeline runs in five stages:
1.2 Data Sources
| Source | Data | Resolution | Auth |
|---|---|---|---|
| Energy-Charts | DA prices (DE-LU bidding zone) | Hourly | None |
| SMARD | Generation by source, consumption, residual load | Hourly | None (CC BY 4.0) |
| Open-Meteo | Wind speed (100m), radiation, temperature, cloud cover | Hourly | None |
Weather Locations
Weather data is fetched from 6 representative locations across Germany, weighted by installed renewable capacity:
| Location | Coordinates | Represents |
|---|---|---|
| North Sea | 54.5°N, 7.0°E | Offshore wind cluster |
| North | 53.5°N, 10.0°E | Onshore wind (Schleswig-Holstein, Niedersachsen) |
| Central West | 51.5°N, 7.5°E | NRW industrial demand, onshore wind |
| Central East | 51.3°N, 12.4°E | Sachsen-Anhalt / Brandenburg wind+solar |
| South West | 48.8°N, 9.2°E | Baden-Württemberg solar |
| South East | 48.1°N, 11.6°E | Bavaria solar |
Wind speed is weighted by installed wind capacity (offshore/onshore splits). Solar radiation is weighted by installed PV capacity (concentrated in southern Germany). Temperature uses equal weighting.
1.3 Features (~30 total)
Weather (D+1 forecast)
- wind_speed_weighted — capacity-weighted wind speed at 100m hub height
- radiation_weighted — capacity-weighted direct solar radiation
- temp_weighted — average temperature across locations
- cloud_weighted — average cloud cover
- wind_north_south_gradient — spatial wind difference (congestion proxy)
- radiation_north_south_gradient — spatial radiation difference
Price Lags (strict no-leakage: D-1 and earlier)
- price_d1_same_hour — same hour yesterday
- price_d7_same_hour — same hour one week ago
- price_d1_mean/min/max — yesterday's daily statistics
- price_7d_rolling_mean/std — 7-day rolling window
Generation Lags (D-1)
- gas_share_d1, coal_share_d1, renewable_share_d1
- consumption_d1, residual_load_d1
- gas_to_coal_ratio — implicit fuel economics proxy (no direct gas/coal/carbon price API)
Calendar
- hour, day_of_week, month
- is_weekend, is_holiday (all 16 German federal states), is_bridge_day
- hour_sin/cos, month_sin/cos — cyclical encodings
Derived
- wind_power_proxy — wind speed cubed (cubic power law, capped at 25 m/s cut-out)
- solar_cf_proxy — radiation × temperature derating (-0.4%/°C above 25°C)
- residual_x_peak — residual load × peak hour interaction
1.4 Model
LightGBM gradient-boosted decision trees with quantile regression objective. Three separate models predict P10, P50 (median), and P90, sharing the same feature set.
A single model covers all 24 hours with hour as a feature, rather than 24 separate hour-specific models. This gives the model more training data per hour and lets it learn cross-hour patterns.
Key hyperparameters
| Parameter | Value | Notes |
|---|---|---|
| n_estimators | 1000 (early stopping) | Stops if validation loss plateaus for 50 rounds |
| learning_rate | ~0.05–0.10 | Tuned via Optuna |
| max_depth | 4–7 | Shallow trees to reduce overfitting |
| num_leaves | 30–63 | Controls tree complexity |
| subsample | 0.6–0.8 | Row sampling per tree |
| colsample_bytree | 0.5–0.8 | Feature sampling per tree |
Training data
Historical data from January 2024 to present (~19,000 hourly samples). Walk-forward evaluation: train on all data except the last 30 days, test on the final 30 days.
Hyperparameters are tuned using Optuna (50 trials) to minimize MAE on the test set. Models are retrained monthly to capture seasonal patterns and structural changes (capacity additions, market reforms).
1.5 Evaluation
Metrics
| Metric | Description |
|---|---|
| MAE | Mean absolute error — average distance between forecast and actual price |
| RMSE | Root mean squared error — penalises large errors more heavily |
| Mean bias | Average (forecast − actual). Positive = systematically overestimating |
| 80% coverage | Fraction of actuals falling within the P10–P90 interval. Target: 80% |
| Skill score | 1 − (MAE / baseline_MAE). Positive = better than persistence (same-hour D-1) |
| Pinball loss | Proper scoring rule for quantile forecasts (lower = better calibrated) |
Baseline
The persistence baseline predicts tomorrow's price = today's price for the same hour. This is the simplest reasonable forecast and the benchmark to beat. A skill score of 0.40 means the model reduces MAE by 40% compared to persistence.
1.6 Timestamp Conventions
Power market timestamps are interval-aligned: 14:00 means the interval from 14:00 to 15:00. All internal timestamps are left-aligned naive UTC (utc_start).
| Source | Native format | Normalisation |
|---|---|---|
| Energy-Charts | Unix seconds (left-aligned) | Direct conversion to UTC |
| SMARD | Unix milliseconds (left-aligned) | ÷1000 to seconds, then UTC |
| Open-Meteo | ISO 8601 in requested timezone | Request in UTC, no conversion needed |
DST transitions are handled explicitly: a CET delivery date may have 23, 24, or 25 hours. The function delivery_date_hours() computes the correct UTC hours for any given Berlin-time date.
2. Grid Poet Image Pipeline
The image pipeline runs hourly and transforms live grid data into AI-generated artwork:
- Fetch — current generation mix, consumption, weather from SMARD + Open-Meteo
- Analyse — Claude AI writes an analysis, poem, and image prompt based on the grid state
- Generate — FLUX.2 Pro creates a painting in the 19th-century German Romantic landscape tradition
- Publish — image + metadata saved, served on the main page
Each image captures a moment in Germany's energy transition — the interplay of wind, solar, fossil fuels, and weather — through the lens of landscape painting.
3. Captured Prices & Cannibalization
The Analytics page shows captured prices and generation-vs-price scatter plots using data from data/da_history.parquet, which is refreshed daily at 02:00 CET.
Data sources (from APIs, stored as-is)
| Column | Source | Unit | Notes |
|---|---|---|---|
| price | Energy-Charts | EUR/MWh | DA price, DE-LU bidding zone |
| wind_onshore | SMARD (4067) | MWh | Hourly generation |
| wind_offshore | SMARD (1225) | MWh | Hourly generation |
| solar | SMARD (4068) | MWh | Hourly generation |
| biomass | SMARD (4066) | MWh | Hourly generation |
| hydro | SMARD (1226) | MWh | Hourly generation |
| natural_gas | SMARD (4071) | MWh | Hourly generation |
| hard_coal | SMARD (4069) | MWh | Hourly generation |
| brown_coal | SMARD (1223) | MWh | Hourly generation |
| consumption | SMARD (410) | MWh | Netzlast (grid load) |
Derived metrics (computed, not from any API)
| Metric | Formula | Notes |
|---|---|---|
| total_generation | Sum of 8 generation sources | Computed in da_forecast/fetch.py during data ingest |
| net_import | max(0, consumption − total_generation) | Positive only during import hours |
| Captured price (per tech) | Σ(priceh × genh) / Σ(genh) | Volume-weighted average DA price, computed daily |
| Baseload price | Σ(priceh) / N | Simple arithmetic mean — the reference line |
4. European Markets
The /markets page compares day-ahead electricity prices, generation mixes, and cross-border flows across 14 European bidding zones: Germany (DE-LU), France (FR), Netherlands (NL), Austria (AT), Poland (PL), Denmark West (DK1), Belgium (BE), Czech Republic (CZ), Spain (ES), Portugal (PT), Norway South (NO1), Sweden Mid (SE3), Finland (FI), and Great Britain (GB).
Data sources
Day-ahead prices for continental zones are sourced from the ENTSO-E Transparency Platform (authoritative TSO data). Generation mix data is sourced from the Energy-Charts API (Fraunhofer ISE, CC BY 4.0). GB uses Elexon BMRS (settlement prices, not DA auction). Data is fetched daily at 03:00 CET and stored locally in parquet files.
Timestamp convention
All timestamps refer to the delivery date — the time interval during which power was physically delivered. Timestamps are left-aligned: "14:00" means the interval 14:00–15:00. All internal storage uses naive UTC. Display conversion to local time is applied per zone.
Source vs derived data
In parquet files, columns without a prefix (e.g., price, wind_onshore) are source data fetched directly from APIs. Columns with a _ prefix (e.g., _total_generation, _renewable_share) are computed/derived values.
ENTSO-E price data notes
DA prices are sourced from ENTSO-E at 15-minute resolution (PT15M) since the SDAC 15-minute go-live. Prices are averaged to hourly for display and storage. Raw 15-minute data is preserved in data/markets/entsoe_raw/ for traceability.
Germany DA prices use the coupled DE-AT-LU domain code (10Y1001A1001A82H), not the standalone DE-LU code. This is because ENTSO-E publishes German DA prices under the historically coupled zone. Austria has its own separate DA prices under 10YAT-APG------L.
All 13 continental zones were cross-validated against Energy-Charts prices for the full Jan 2024 – Mar 2026 period: 99.9–100% of hours match within 0.01 EUR/MWh. One known exception: AT on 26 June 2024 shows Energy-Charts anomalies (up to 1,965 EUR/MWh) that ENTSO-E does not have — Energy-Charts data error, not an ENTSO-E issue.
Cross-border flows
Cross-border data shows scheduled commercial exchanges for Germany. Positive values = export from DE, negative = import to DE. Source: Energy-Charts /cbet endpoint (values in GW).
5. Data Attribution
- SMARD — Bundesnetzagentur, licensed under CC BY 4.0
- ENTSO-E — Transparency Platform, DA prices for 13 continental zones
- Energy-Charts — Fraunhofer ISE, generation mix data (CC BY 4.0)
- Open-Meteo — open-source weather API using ECMWF, DWD, and other NWP models
- German holidays — computed using the holidays Python library