Seeing inside async Rust: Tokio, tracing, tokio-console and OpenTelemetry in production

# The Problem

Your Rust service is fast — except when it isn’t. A few times an hour, p99 latency jumps from 8ms to 900ms for no reason anyone can find. The logs show nothing unusual. CPU is fine. There’s no error. Requests just… hang, briefly, and then recover, and by the time you’ve SSH’d in it’s over.

You take the one tool you trust — a stack trace — and it’s useless. It shows a handful of Tokio worker threads sitting in the runtime’s poll loop. That’s it. It cannot tell you that somewhere among the ten thousand async tasks your service is juggling, one of them made a blocking call that parked a runtime worker and starved everything else for 300 milliseconds. Async Rust doesn’t hide the bug in a hard-to-read place; it hides it in a place a stack trace fundamentally cannot see.

This is the tax of an async runtime: enormous concurrency, and near-zero visibility into it by default. The fix isn’t heroics — it’s instrumentation. And in 2026 the tooling to see inside async Rust is finally production-grade.

# The Three-Layer Strategy

There’s no single “observe async Rust” switch. There are three layers, each answering a different question, and you want all three:

1. tracing — structured, causality-aware application telemetry. Spans and events that follow a request through your async code. Answers “what is my code doing?”
2. tokio-console — runtime-level introspection. Live task states, poll times, wakeup counts, and the smoking gun for blocked or starved tasks. Answers “what is the runtime doing?”
3. tracing-opentelemetry — export spans and metrics to your existing OTel collector, so Rust services land in the same dashboards as everything else. Answers “how does this fit the rest of my system?”

# The Architecture Perspective

The architectural insight is that async Rust separates tasks (units of logical work) from threads (units of execution) — and your observability has to follow the tasks, not the threads. A request isn’t pinned to a thread; it’s a task that gets polled, parked on I/O, moved to another worker, and resumed. Thread-centric tooling (the stack trace, thread CPU%) measures the wrong axis.

flowchart LR
    APP["Async Rust service"] --> T["tracing<br/>spans/events"]
    APP --> RT["tokio runtime<br/>instrumentation"]
    T --> OTEL["tracing-opentelemetry<br/>→ OTLP"]
    RT --> CON["tokio-console<br/>consumer"]
    OTEL --> COLL["OTel Collector"]
    COLL --> DASH["Shared dashboards<br/>/ traces"]
    CON --> OPS["Live task states,<br/>wakeups, blocked tasks"]

    classDef app fill:#2563eb,color:#fff,stroke:#1e40af;
    classDef inst fill:#059669,color:#fff,stroke:#065f46;
    classDef out fill:#d97706,color:#fff,stroke:#92400e;
    class APP app;
    class T,RT,OTEL,CON inst;
    class COLL,DASH,OPS out;

The second architectural payoff is one pane of glass across languages. Because tracing-opentelemetry speaks OTLP and propagates W3C trace context, a Rust service sits in the same tracing backend as your JVM, Go, or Node services, with correlated trace IDs across service hops. If you’ve already built an OpenTelemetry collector and dashboards for other stacks, Rust plugs into it — you don’t run a separate, Rust-shaped observability island.

# The Developer Perspective

Start with tracing. Annotate the functions you care about with #[instrument] and emit structured events — spans give you async-aware causality that log lines can’t:

use tracing::{instrument, info};

#[instrument(skip(db), fields(user_id = %req.user_id))]
async fn handle_checkout(req: CheckoutRequest, db: &Db) -> Result<Receipt> {
    info!("checkout started");
    let order = place_order(&req, db).await?;   // child spans nest automatically
    charge(&order).await
}

Next, wire up tokio-console to see the runtime. It needs the tokio_unstable cfg and the console subscriber:

// main.rs — enable the console subscriber (dev/staging, or gated in prod).
console_subscriber::init();

// Build with the unstable flag so task instrumentation is compiled in:
//   RUSTFLAGS="--cfg tokio_unstable" cargo run

Now the classic bug becomes visible. Reproduce a blocking call on the async runtime — the thing your stack trace couldn’t see:

// THE BUG: a synchronous, CPU-heavy call directly on an async task.
async fn resize(image: Image) -> Image {
    heavy_cpu_resize(image)      // blocks the worker → starves other tasks
}

// THE FIX: move blocking work off the runtime.
async fn resize(image: Image) -> Image {
    tokio::task::spawn_blocking(move || heavy_cpu_resize(image))
        .await
        .unwrap()
}

In tokio-console, the buggy version shows a task with a huge busy duration and a starved poll loop — the exact task that a stack trace rendered invisible. The fix frees the worker, and the console confirms it.

Finally, export to OpenTelemetry so it all lands in your shared backend:

use tracing_subscriber::prelude::*;

let otel = tracing_opentelemetry::layer().with_tracer(tracer);   // → OTLP
tracing_subscriber::registry()
    .with(otel)
    .with(tracing_subscriber::fmt::layer())
    .init();

# The DevOps Perspective

Instrumentation is a production commitment with real trade-offs — treat it like one.

tokio-console is observability, not just a local toy. The 2026 shift is teams running it against remote instances as part of their monitoring stack, not only on a laptop. You can connect to instrumented production instances to watch live task states and wakeup counts — but budget the overhead and gate it deliberately (it’s not free, and it needs tokio_unstable compiled in). Decide up front whether it’s always-on in staging and on-demand in prod, or always-on everywhere with sampling.

flowchart TD
    Q1{Where does it run?} --> DEV["Dev/staging:<br/>always-on tokio-console"]
    Q1 --> PROD["Prod:<br/>on-demand / gated"]
    DEV --> OTEL["tracing → OTLP → collector<br/>(always-on, sampled)"]
    PROD --> OTEL
    OTEL --> ONE["One dashboard,<br/>correlated across services"]

    classDef ask fill:#2563eb,color:#fff,stroke:#1e40af;
    classDef env fill:#059669,color:#fff,stroke:#065f46;
    classDef out fill:#d97706,color:#fff,stroke:#92400e;
    class Q1 ask;
    class DEV,PROD env;
    class OTEL,ONE out;

Sampling and overhead are your levers. Full tracing on every request has a cost. Use head or tail sampling for the OTLP export, keep high-cardinality fields in check, and measure the instrumentation’s own footprint under load — the goal is visibility that survives production traffic, not a debug build shipped by accident.

Correlate with the rest of the fleet. Propagate W3C trace context across service boundaries so a Rust hop shows up as one continuous span alongside your other services. If you already operate an OTel collector for JVM or Go workloads, point Rust at the same one — same dashboards, same alerting, correlated trace IDs.

# Putting It Together

Async Rust buys you enormous concurrency and charges you in visibility. The default state is a black box that a stack trace can’t crack. The three-layer strategy — tracing for application spans, tokio-console for the runtime, and tracing-opentelemetry to land it all in your shared backend — turns that black box into something you can tune, capacity-plan, and debug. The tooling is production-grade now; the blocking-call-that-starves-the-runtime bug that used to cost you an afternoon becomes a task with a suspicious busy-duration you can spot in seconds.

# References

• Tokio Console in Production: From Debugging Tool to Observability — Rustaceans
• Getting started with Tracing — Tokio docs
• Rust Observability: Logging, Tracing, and Metrics with OpenTelemetry and Tokio — dasroot
• Rust Async Programming with Tokio: A Practical Guide for 2026 — Dev Note
• tokio-rs/tracing — GitHub
• tokio-console — docs.rs