Skip to content
Bryan Brkic
All work

Pelmorex Corp — The Weather Network

Real-time precipitation alerts, from forecast to phone

A precipitation alert has a short shelf life. If it lands after the rain starts, it is not an alert, it is a notification about something you already noticed. As tech lead, I architected and delivered the platform that keeps that from happening across Canada and the United States.

Visit theweathernetwork.com
Role
Tech lead — architecture and delivery
Team
Two developers and one developer-in-test
Period
Apr 2023 — Sep 2023
Core stack
Python, TypeScript, AWS, GDAL/rasterio, Terraform

The problem

Coverage makes it hard

Serving one city is a scheduling problem. Serving every populated area in Canada and the United States is a fan-out problem. The forecast keeps updating, each subscriber has their own saved locations, and the work created by a single update varies by orders of magnitude depending on where the weather is.

The point of the system is precision and lead time. Every check is scoped to a small area — a few kilometres across — around a subscriber's saved location, and a notification means precipitation is on its way, typically some tens of minutes out rather than already falling. That window is only useful if it survives the trip from forecast to phone. Everything from a forecast update to a phone buzzing has to finish well inside ten minutes, nationwide, and often much faster than that — a quiet day with little precipitation to process clears in a couple of minutes.

The path

What happens between the forecast and the phone

  1. Stage 1

    Forecast lands

    A new short-range forecast lands in object storage and a router picks it up, tagging the job so the same piece of work can be followed through every hop that follows.

  2. Stage 2

    Masking

    Geospatial tooling turns the forecast into a mask of where precipitation is actually expected, reprojecting the source grids onto one common resolution and trimming out the areas that should never raise an alert.

  3. Stage 3

    Tiling

    The mask is cut into fixed-size tiles, and a tile with nothing in it is dropped before it ever reaches a queue — there is nothing downstream worth doing.

  4. Stage 4

    Querying

    Each remaining tile still stands for thousands of points to check. A fleet that autoscales on queue depth works through them in batches against the forecast API, applying the intensity rules that decide who actually gets notified.

  5. Stage 5

    On the phone

    Notifications go out as push topics, batched and retried, in English and French, on every app this platform serves.

Coast to coast, well inside ten minutes — faster on quiet days

Architecture

The same hand-off, three times

Three different services do the real work — masking, tiling, and notifying — but they never call each other directly. Each one drops its output in S3, a router Lambda picks it up and queues it, and the next service in line reads from that queue on its own schedule. The hand-off is identical every time, which is what makes the chain easy to reason about even though it runs several hops deep.

Real-time precipitation notification pipelineA forecast lands in object storage and triggers a router that queues it. A masking service, a tiling service and a notification service each pick up the same storage-to-router-to-queue hand-off in turn, before the result is handed to a push service and out to devices.S3Forecast landsλ → SQSRouter, per-hop queueMask serviceGeospatial maskingλ → SQSRouter, per-hop queueTiling serviceEmpty tiles droppedλ → SQSRouter, per-hop queueNotify serviceAutoscaled fleetPush serviceBatches topicsFCMPush to the phone
Forecast to phone — the same S3 → Lambda → SQS hop repeats three timesScroll the diagram sideways →

Scale

Why tiles, why spot instances

The naive version of this pipeline checks every point on a continent-sized grid on every forecast update. That does not fit in the window. Two decisions made it fit.

  • Sparse tiling: a single tile stands for thousands of potential API calls, but the tiling service only passes on tiles that actually contain precipitation. Most of a mask, most of the time, goes nowhere.
  • Demand-matched compute: the notification fleet runs on spot instances behind an Auto Scaling group tuned to queue backlog, so a light-precipitation day and a coast-to-coast system both stay well inside the delivery window.
  • The same discipline applies downstream: location lookups and push sends are both batched, so nothing waits on a single slow call.

Precision

Not every update is worth a buzz

Speed and precision only matter if the alert is actually useful. A subscriber who gets buzzed every time the forecast shifts by a fraction of a millimetre learns to ignore the app, so the notification logic has its own rules for when a new update is worth interrupting someone over.

  • Escalation always notifies: if a forecast for moderate rain becomes heavy rain, both alerts go out, because a worsening forecast is new information worth acting on.
  • De-escalation is suppressed: the reverse — heavy rain settling back to moderate — does not trigger a second, less urgent alert. Nobody needs to be told the weather got slightly less bad.
  • Location changes are handled on-device: if a subscriber already has a moderate-rain alert and then moves somewhere with the same class of alert pending, the app suppresses the repeat rather than re-notifying for something they've effectively already been told. The backend sends the data that makes that comparison possible.

Next case study

Systems I helped scale at The Weather Network

Read it

Get in touch

If any of this is the kind of problem you are hiring for, I would like to hear about it.

brkicb1@gmail.com