Freight rates move. Reports about freight rates mostly don't. My EU Freight Rate Risk Monitor started as the second kind: a published study on 20+ years of Eurostat data, quantifying how differently air and road rates behave. Air runs about 1.6x road's volatility inside a 6.7% annual band, peaks in Q2, and rewards trough-season tendering in 75% of 85 backtested market-years. Useful numbers. Quarterly numbers.
Logistics operations don't work in quarters. A tender desk finds out about a rate spike when a carrier declines a load, not when Eurostat publishes. So I rebuilt the monitor as an event-driven system.
What runs now: the study's real index series replays as a stream of rate-quote events through Kafka topics. Tick-level movement between the quarterly anchors is simulated, calibrated to each market's published volatility, and labeled as simulation. One feed is fully live: the ECB daily euro reference rate. A consumer scores every event against a rolling window using the same z-score method the study applied at rest. Anomalies become alert events on their own topic, land in a DuckDB sink, and fire HMAC-signed webhooks to subscribers. Inbound works too: a FastAPI endpoint accepts carrier rate updates and normalizes them onto the same stream. Bad events go to a dead-letter queue instead of disappearing.
The satisfying part: replaying 2019-2022 through the pipeline, the 2020 air-freight shock fires breach alerts exactly where the study's charts show the spike. The batch analysis and the streaming system agree, because the second is the first, set in motion.
Numbers from my machine: the 2019-2022 replay pushes 2,136 events through at 212 events per second sustained, produce-to-alert latency 121 ms at the median (p95 1.6 s, dominated by consumer-group startup). 51 breach alerts fire on the replay, 27 of them in 2020 with air lanes on top. The same pipeline replayed over quiet 2015-2018 fires 2. Unthrottled, the consumer drains the full replay at 479 events per second. The simulation is seeded: every run, on every machine, produces the identical alert set.
Why this project: my work sits where freight procurement meets platforms - 12+ years of pricing and tendering, 3.5 of them configuring a SaaS tender platform and its EDI/API/webhook connections for 68+ enterprise shippers. Event-driven architecture is where logistics platforms are heading; agents reacting to live signals need pipelines like this underneath. Building one end to end, on my own data, seemed like the honest way to learn its edges.
Repo: github.com/khareparag/freight-rate-stream (Docker Compose, runs in 10 minutes)
Study + live dashboard: github.com/khareparag/freight-rate-risk-monitor
open data, open method, reproducible end to end.