A distributed job scheduler, part 3: measuring before scaling

The spec was explicit: add a Redis “hot index” of due runs so the claim can pop work from Redis instead of scanning Postgres. It’s a reasonable-sounding optimization, the kind that ends up in a design doc because someone once saw a SELECT show up in a slow-query log. There was just one problem: nothing in the system had ever been measured. The claim scan might be the bottleneck. It might be three percent of the work. Nobody knew, and the design was about to add a second datastore on a hunch.

So before building the cache, I built the thing that could tell me whether to build the cache. This post is about that — measuring a distributed scheduler before scaling it, and the four scaling mechanisms that followed once there were numbers to reason from. Part 1 designed the claim; part 2 built it in Rust; this is where it grows up.

# First, numbers

A benchmark that measures the wrong thing is worse than none, because it produces a figure people quote. So the bench drives real Postgres and NATS in containers, reports its own environment, and ends every run with an explicit statement of what the number does not mean. On a four-core laptop with the database sharing the host with the load generator — a floor for this setup, not a production ceiling — draining a standing backlog through the claim path came out around this:

There’s the answer, and it’s not subtle. The SELECT … FOR UPDATE SKIP LOCKED claim — the exact mechanism a Redis hot index would replace — is a rounding error in the loop. Ninety-seven percent of the time is one synchronous publish round-trip per run, and a Redis due-index touches none of it.

# Then build the cache anyway — and let the bench judge it

Here’s the part I think is worth the whole exercise. Even with the evidence saying no, there’s value in building the thing and measuring it, rather than either skipping it silently or asserting it wouldn’t help. So I built the Redis hot index — as a strict hint, never a source of truth. Postgres stays authoritative: the engine pops candidate ids from a Redis sorted set, then claims exactly those ids in Postgres under SKIP LOCKED. A stale id (already claimed) matches nothing and is dropped; a wiped index just sends the engine back to the Postgres scan. Three integration tests pin that invariant — the hot path agrees with the scan, FLUSHDB loses nothing, a stale hint is never re-claimed.

Then the bench ran both paths against the same backlog, claim only, publishing excluded so the comparison isolates the claim mechanism:

The hot index is ~21% slower. It adds a pop_due round-trip per batch and still does the same claim write, so it can only pay off if the scan it replaces were the bottleneck — and the scan is ~3% of the loop. It is wired into nothing by default; the engine still claims via the scan.

I find this a genuinely useful outcome. “We built it, measured it, and it hurt” is a stronger answer than either “we skipped it” or “we added it because the spec said so.” The reference value is the method — build it, prove it correct, measure it, record the verdict — not the cache. The measurement that would justify Redis, and which a laptop bench can’t produce, is many concurrent claimers against a non-containerised database, where SKIP LOCKED contention would first appear. Until that’s measured, the pieces are there, tested, and turned off.

# The scaling that the numbers did support

With a bench in place, the rest of Phase 3 became decisions backed by evidence rather than instinct.

Partition job_runs by time. At the design target, the runs table grows without bound, and the growth story can’t be a DELETE that scans and writes tombstones across the live index the claim depends on. So the table is PARTITION BY RANGE (scheduled_at), one partition per day: the due scan prunes to recent partitions, and ageing out old runs is a partition DROP — one metadata operation, not a scan. Postgres makes you put the partition key in every unique constraint, so the primary key became (id, scheduled_at); a DEFAULT partition guarantees no insert ever fails for want of a range. The proof it’s transparent: every existing claim test passes unmodified against the partitioned table.

Bound one tenant’s share of a claim. A tenant that falls far behind — a thousand overdue runs, all older than everyone else’s — would win every oldest-first batch and starve every other tenant. So the claim takes a per_tenant_cap: a ROW_NUMBER() OVER (PARTITION BY tenant ORDER BY scheduled_at) ranks each tenant’s due runs, and only the first cap per tenant are eligible before the overall limit applies. A detail that matters: the window function lives in its own CTE, because Postgres forbids FOR UPDATE in a query that uses one — so ranking and locking are two separate steps. And it’s honest about what it is: a per-batch cap, not a token-bucket rate limiter. It stops starvation; it isn’t a requests-per-second limit, and the docs say so rather than overclaiming.

Autoscale on a metric that means something. The Kubernetes manifests give the worker a plain CPU HPA, but the interesting one is a KEDA ScaledObject that scales the engine on due_lag_seconds — the histogram of how late runs are when claimed. Runs getting later means the due queue is backing up faster than one dispatcher drains it, and more claimers is the right response; SKIP LOCKED makes them safe. Per-replica lease-owner uniqueness comes free, too: leave SCHEDULER_OWNER unset and the config falls back to HOSTNAME, which Kubernetes sets to the pod name.

Trace a run across the process boundary. Metrics tell you how many and how fast; a trace tells you where one run’s time went — and for a scheduler, that journey crosses the NATS message between dispatch and execution, which no in-process tracer sees. So the dispatcher injects its span’s W3C traceparent into the message headers and the worker extracts it and parents its execution span to it. Dispatch and execution end up on one trace instead of two unrelated ones. Export is opt-in via OTEL_EXPORTER_OTLP_ENDPOINT; the context propagation itself is free and always on, so it’s never a silent no-op.

# Putting it together

The thread running through all of it is one habit: measure, then decide. The Redis story is the sharpest version — a spec’d optimization that the evidence said not to build, built anyway to prove the evidence, and confirmed at ~21% slower. But it’s the same habit behind partitioning by the key the scan actually uses, capping fairness where the starvation actually happens, and autoscaling on the lateness metric that actually signals backpressure. None of those are guesses. They’re responses to numbers.

The design target — 100M runs — remains designed-for and unverified, and the docs say exactly that. That’s the last piece of the habit: a laptop bench is a floor, not a ceiling, and calling it anything else would undo the point of measuring in the first place.

# References

• PostgreSQL declarative partitioning: https://www.postgresql.org/docs/current/ddl-partitioning.html
• ROW_NUMBER() window function: https://www.postgresql.org/docs/current/functions-window.html
• KEDA (Kubernetes event-driven autoscaling): https://keda.sh/
• W3C Trace Context: https://www.w3.org/TR/trace-context/
• OpenTelemetry OTLP exporter: https://opentelemetry.io/docs/specs/otlp/