A distributed job scheduler, part 2: building it in Rust

The test suite was green. Every claim test, every materializer test, every delivery test — passing. Then I ran the engine against a real Postgres for two minutes with a five-second job, and it produced 888 runs for a schedule that should have produced about twenty-four. The materializer was manufacturing duplicate work as fast as it could poll, and not one test had noticed.

That gap — between a green suite and a system that works — is what this post is about. Part 1 covered the design of a distributed scheduler: the claim, the lease, the honest delivery guarantee. This one is about building it in Rust, and specifically about the discipline that turns “the tests pass” into “the thing is actually correct,” because those are not the same sentence.

# Ports as traits, adapters as crates

The domain defines ports — traits for the things it needs done — and knows nothing about who implements them. RunRepository is the interesting one: it claims due runs, releases them, reclaims expired leases, records completions. In Rust these ports are traits with native async fn (return-position impl Trait in traits — RPITIT), and the application services are generic over them:

pub trait RunRepository: Send + Sync + 'static {
    fn claim_due(&self, now: OffsetDateTime, limit: i64, owner: &str, per_tenant_cap: i64)
        -> impl Future<Output = DomainResult<ClaimOutcome>> + Send;
    // insert_runs, release, reclaim_expired, complete, ...
}

No #[async_trait], no Box<dyn>. The services (ClaimAndDispatch<R, P, C>) are monomorphized over their ports, so calls are static dispatch — no vtable, no indirection. The Postgres adapter is one crate implementing the trait; the in-memory fake is another.

There’s a real cost to name, because pretending otherwise would be the first dishonest sentence in the codebase: RPITIT ports are not object-safe. Nothing can hold a Box<dyn RunRepository>. Every consumer is generic over the port, which ripples into type signatures everywhere. That’s the trade — zero-cost static dispatch in exchange for never being able to erase a port behind a trait object. Worth it here; worth knowing before you commit to it.

One exception proves the rule. When I later added a metrics port, I made it synchronous and object-safe on purpose — recording a metric must not be an await point on a hot loop, or the instrumentation changes the thing it measures — so the use cases hold an Arc<dyn Metrics> rather than threading yet another generic through every construction site. The shape of a port should follow what it’s for, not a blanket rule.

# The bug the tests couldn’t see

Back to the 888 runs. The materializer’s job is to look at a recurring job and propose the runs due within a horizon. The instants it proposes have to be a function of the job — anchored at the job’s created_at, so a run always lands on created_at + k × period. Two ticks a second apart then propose overlapping sets, UNIQUE (job_id, scheduled_at) collides on the overlap, and ON CONFLICT DO NOTHING absorbs the duplicates. The row count stays flat.

The bug: it anchored the grid on clock.now() instead. Since now moves every tick, every tick walked a different grid, the proposed timestamps never repeated, the unique constraint never collided, ON CONFLICT absorbed nothing, and a job accumulated a fresh horizon of runs every poll interval. Unbounded growth.

Why did every test miss it? Because every materializer test used a FixedClock. Under a clock that never moves, a grid anchored on now and a grid anchored on created_at are indistinguishable — they produce identical output. The tests weren’t wrong, exactly; they were vacuous for this property. The fixture made the bug unobservable.

This is why the discipline matters more than the coverage number. Three separate vacuous tests shipped green in this project, each because a fixture made the property it claimed to check impossible to observe.

# Fakes that mirror the adapter, or they lie

Because the ports are traits, the application layer tests against an in-memory RunRepository. The temptation is to make that fake permissive — return something plausible and move on. That temptation is a trap: a fake that behaves differently from the real adapter lets tests pass against production code that’s broken.

So the in-memory RunRepository mirrors the Postgres adapter on every semantic that matters. It records a lease expiring LEASE_SECS after now, so reclaim tests have something real to read. It increments attempt on claim, so the attempt cap is actually reachable through the fake. It buries a run that has exhausted MAX_ATTEMPTS as dead and excludes it from the batch — exactly like the SQL — so the “past its cap” tests aren’t checking a behaviour only the database has. Every one of those mirrorings has a comment explaining which adapter behaviour it reproduces and what test would silently pass against a broken adapter if it didn’t.

A fake that’s easier than the real thing isn’t a test double. It’s a way to avoid testing.

# Mutation testing as the acceptance bar

Green tests prove the tests pass. They don’t prove the tests would fail if the code were wrong — and a test that can’t fail is decoration. So the standard I held every mechanism to was: remove the mechanism, and the specific test that covers it must fail.

Concretely: to check that the per-tenant fairness cap actually works, I don’t just assert the happy path. I break the ranked filter in the claim SQL so one tenant can dominate again, run the full suite, and confirm the fairness test — and only the fairness test — goes red. Then I restore it. If breaking the mechanism leaves the suite green, the test was measuring nothing and gets rewritten.

There are two failure modes of this technique that are easy to fall into, and both bit me at least once:

• A mutation that makes a test hang hasn’t been checked. A timeout isn’t a clean failure; it’s ambiguity. The drain test uses a paced runtime and a bound specifically so that removing the timeout mechanism fails rather than hangs.
• Never run the mutation under a test-name filter. cargo test complete silently skips completing_a_run, so a filtered run can report “passed” for a test that never executed. The mutation has to run against the full, unfiltered suite.

# When the honest fix is a small refactor, not a workaround

Two examples of the code changing shape to stay correct, rather than being patched around.

The first: counting runs that get buried (abandoned after exhausting their attempts). Burial happens inside the claim SQL — a due run past its cap is moved to dead and excluded from the batch — so no later stage can see it, and the metric that should count it had nothing to observe. The wrong fix is a second query racing the first. The right one was to have the claim return a small ClaimOutcome { claimed, buried } value instead of a bare list, so the buried count rides back with the claimed runs. Making that survive a pass that buries runs but claims none meant reshaping the SQL into a tagged UNION ALL — because the old shape returned only claimed rows, so an all-buried tick came back empty and the count vanished. A subtle correctness edge, fixed in the query rather than papered over above it.

The second, smaller: the ClaimOutcome derefs to the claimed slice (impl Deref<Target = [JobRun]>). Roughly fifty call sites that only wanted the runs kept compiling unchanged — .len(), .iter(), indexing all still work — while the handful that report metrics reach for .buried. The refactor’s blast radius stayed small because the new type was designed to be transparent to the callers that didn’t care.

# Putting it together

Building this in Rust was less about the language’s guarantees — though Send + Sync bounds and exhaustive match caught real things — and more about a discipline the language doesn’t enforce for you. Ports as zero-cost traits keep the domain clean and testable. Fakes that mirror the real adapter keep the tests honest. And mutation testing keeps the tests load-bearing, so “green” means “this would have caught the regression,” not “these assertions happened to run.”

The 888-run bug is the one I keep coming back to, because it’s the cheapest possible reminder: a green suite is a claim about your tests, not about your system. The two only agree if you make them.

Part 3 is about scaling — and about resisting the urge to scale something you haven’t measured. It opens with a cache that everyone wanted and the numbers that said don’t build it.

# References

• async fn in traits (RPITIT), stabilized: https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits.html
• PostgreSQL INSERT … ON CONFLICT: https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT
• Mutation testing, the idea: https://en.wikipedia.org/wiki/Mutation_testing
• cargo mutants (the tool that formalizes this): https://mutants.rs/