Skip to content

Keep requests through middleware in a single OpenTelemetry trace - #100136

Merged
sokra merged 5 commits into
canaryfrom
fleet/otel-single-trace-middleware
Oct 15, 2026
Merged

sokra merged 5 commits into
canaryfrom
fleet/otel-single-trace-middleware

Conversation

@sokra

@sokra sokra commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

What?

Requests that pass through middleware / proxy.ts now produce one OpenTelemetry trace, instead of two or three.

Before, with OTEL configured in instrumentation.ts:

Request No outer span With an outer span (APM, custom server)
Middleware continues to render (next() / rewrite) 3 traces (2 before 16.3) 2 traces (1 before 16.3)
Middleware redirect / direct response 2 traces (1 before 16.3) 2 traces (1 before 16.3)

After: 1 trace in every case.

Why?

Two independent causes:

  1. Long-standing (13.5 → canary), Next.js middleware tracing produces two traces per request #91282. When middleware lets a request continue, the router server runs BaseServer.handleRequest twice: a middleware-invoke pass, then a render pass. Each pass extracts its context from the incoming request headers. Without an incoming traceparent, each pass started its own root trace.
  2. 16.3 regression from otel: Use correct parent span for Node.js middleware #95306. In-process Node.js middleware ran under ROOT_CONTEXT to make Middleware.execute a sibling of handleRequest. That also detached it from any outer span, so the middleware span always started an extra trace.

Customers upgrading 16.2 → 16.3 see this as "doubled traces".

How?

When middleware lets a request continue, the render is always nested inside the middleware span, in every host:

next start / next dev:
 handleRequest GET (bubble, root)
  └─ middleware GET
      └─ GET /page

separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent:
 middleware GET (root)
  └─ GET /page

host span / incoming traceparent P:
 P
  └─ middleware GET            (next start: P → handleRequest GET → middleware GET)
      └─ GET /page

Middleware → render. The middleware adapter forwards the Middleware.execute span's own W3C context (traceparent / tracestate, via the configured propagator) as request-header overrides on next() and rewrite responses. Every host already applies these overrides to the downstream request, including the next start router server, so no new host protocol is needed.

  • Override lists replace the whole downstream header set. So:
    • No user overrides: the adapter lists every request header the middleware received, plus the trace headers.
    • User overrides present: they are kept, and the trace headers are added.
    • User-set trace headers: a traceparent / tracestate the middleware explicitly set (different from the one it received) wins. A value copied unchanged from the request, as with new Headers(request.headers), is replaced so the render still nests.
  • Redirects and direct responses are untouched.

Request → middleware (self-hosted). Once the middleware-invoke pass's handleRequest span is active, its W3C context is written onto the request headers. The middleware adapter (Node.js and the Edge sandbox) extracts it, so the middleware span is a child of that pass. The middleware-pass span keeps its existing name (GET); the route stays on the render-pass span. If there is no middleware span to forward (for example OTEL is registered only for Node.js while middleware runs on Edge), the render falls back to this context and still stays in one trace.

#95306 regression. The ROOT_CONTEXT detachment is removed, so an outer span, an incoming traceparent, or the middleware-pass span now parents Node.js middleware normally.

All of this only runs when OpenTelemetry is active and a valid span context exists. Without OTEL, no headers are added.

One-time initialization traces stay separate by design: instrumentation registration, route-module prewarming, and the once-per-process lazy RouteModule.prepare of direct entrypoints.

Tests

  • New deploy-capable suite test/e2e/app-dir/otel-middleware-single-trace. It runs Node.js proxy.ts and Edge middleware.ts with @vercel/otel. Middleware reports its active trace and span IDs; the render reports its active trace ID and the traceparent it received. The test asserts one trace and that the render's parent is the middleware span, so no collector is needed. It runs in dev, start and deploy mode, with no deploy gate. It covers:
    • continue to a page and to a route handler, plus rewrite, redirect and a direct response;
    • a request the middleware doesn't match, and an incoming traceparent;
    • user header overrides, including a deletion (the render still nests), and a user-set traceparent;
    • no internal x-middleware-* headers reaching the client.
  • Existing collector suite test/e2e/opentelemetry/instrumentation. The middleware tests now assert real trace IDs and the exact shape: the render span's parent is the middleware span, the middleware span's parent is the middleware-pass handleRequest (self-hosted) or the external / host parent. They also check every span exported for the request.
    • Setups: default and direct-entrypoint hosting; Edge and Node.js middleware; root context and incoming traceparent.
    • Requests: page, route handler, rewrite and redirect.
    • New variant: direct entrypoints with a host parent span.
  • Unit tests for the override helper (forward-trace-context.test.ts), including unchanged copied trace headers.

Verification

  • The new suite's nesting checks and the collector shape checks fail before this change (render parented to handleRequest or the external span) and pass after it.
  • New suite passes in dev (Turbopack), start (Turbopack, webpack), and the same with __NEXT_CACHE_COMPONENTS=true.
  • Full test/e2e/opentelemetry/instrumentation/ passes in start mode (Turbopack); the middleware tests also pass in dev (Turbopack) and start (webpack).
  • Passed on the previous revision (not re-run for the nesting change): test/e2e/opentelemetry/client-trace-metadata, test/e2e/app-dir/otel-parent-span-propagation, test/e2e/middleware-request-header-overrides, and the W3C trace-header fetch-cache test in test/e2e/app-dir/app-static.
  • Deploy: an earlier revision's trace-ID assertions passed on Vercel (default and Cache Components). The new parent-span assertion runs on Vercel for the first time in this revision's CI.

Fixes #91282

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Tests Passed

Commit: 5d1fa1a

vercel-fleet-prod Bot and others added 2 commits October 13, 2026 16:33
…xture

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
@sokra
sokra requested a review from lukesandberg October 14, 2026 08:26

sokra commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

Reviewed 8ef9202

@sokra
sokra marked this pull request as ready for review October 14, 2026 11:00
Comment thread packages/next/src/server/base-server.ts Resolved
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
@sokra
sokra merged commit 2e55993 into canary Oct 15, 2026
120 checks passed
@sokra
sokra deleted the fleet/otel-single-trace-middleware branch October 15, 2026 06:56
sokra added a commit that referenced this pull request Oct 15, 2026
…ry trace (#100226)

Backports #100136 to `releases/lts/active`.

<!-- NEXT_JS_LLM -->

<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 16, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 17, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 19, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic added a commit that referenced this pull request Oct 19, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra added a commit that referenced this pull request Oct 21, 2026
…race" (#100136)

This reverts commit f1a69d7 (#100302).

Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic added a commit that referenced this pull request Oct 21, 2026
…00294)

## Summary

Stacked on #100349 (request span before route module preparation).

Relands #100136 (reverted in f1a69d7) and changes the span shape so
middleware and the route are **siblings** in one trace per request.

### Resulting shapes

```
next start / next dev:
 handleRequest GET (bubble, root)
  ├─ middleware GET
  └─ GET /page

next start / next dev with host span / incoming traceparent P:
 P
  └─ handleRequest GET (bubble)
      ├─ middleware GET
      └─ GET /page

separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent:
 GET (Next.request, root, inserted by the middleware adapter)
  ├─ middleware GET
  └─ GET /page

separately-invoked middleware with host span / incoming traceparent P:
 P
  ├─ middleware GET
  └─ GET /page
```

### How

- **Self-hosted:** next-server's middleware-invoke `handleRequest` pass
writes its W3C context onto the request (`traceparent`, and `tracestate`
only together with it: a received `tracestate` is removed when the
context has none). Middleware (Node in-process, or the Edge sandbox via
the header) and the render pass both continue it. The render pass forces
that extraction even under an active host span (custom server / APM), so
it nests under the bubble span; a different `traceparent` the middleware
explicitly set still wins.
- **Middleware adapter:** an active host span wins over a request
`traceparent`; otherwise the request's `traceparent` is continued, and
the middleware never changes it. Without any valid context (and OTEL
enabled) the adapter inserts a `Next.request` SERVER span (new
`NextSpan` enum) named after the method (`http.target` keeps the path).
When the downstream request doesn't already carry that shared parent,
its `traceparent`/`tracestate` are forwarded via the existing
`x-middleware-request-*` overrides, always as a pair: a user-set
`traceparent` keeps exactly the trace headers the user set. No headers
are added without OTEL.
- Request-header overrides replace the whole header set, so when the
user set no overrides, the received headers are listed alongside the
trace headers. There's no API to add a single header yet; that needs a
server and platform change.
- Node middleware is no longer detached via `ROOT_CONTEXT` (the 16.3
regression from #95306).

The inserted `Next.request` root ends when the middleware finishes, so
the render child (a separate invocation) can outlive it.

## Verification

- Trace shapes are asserted as printed span trees
(`test/lib/otel-span-tree.ts`, from #100349) in inline snapshots. Span
IDs are normalized away, but distinct traces keep distinct labels, so a
split trace shows up as `[trace A]` / `[trace B]`.
- `test/e2e/app-dir/otel-middleware-trace` (added in #100349 with the
previous behavior): this PR's snapshot diff shows middleware and render
becoming siblings in one trace, in dev, start, Cache Components and
deploy, for Node.js and Edge middleware and behind a custom server with
a host span (including an incoming `tracestate` not being paired with
the host span's context).
- Collector suite `test/e2e/opentelemetry/instrumentation`: the
middleware test asserts the full span tree in next start/dev, direct
entrypoints with/without host span, and the custom server (host span,
user-set `traceparent`).
- Unit tests for the tracer and `forwardTraceContextToRequest` changes.
- `test/e2e/middleware-request-header-overrides`,
`test/production/typescript-basic`,
`test/production/middleware-typescript` pass.

<!-- NEXT_JS_LLM -->


<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic pushed a commit that referenced this pull request Oct 21, 2026
…00294)

## Summary

Stacked on #100349 (request span before route module preparation).

Relands #100136 (reverted in f1a69d7) and changes the span shape so
middleware and the route are **siblings** in one trace per request.

### Resulting shapes

```
next start / next dev:
 handleRequest GET (bubble, root)
  ├─ middleware GET
  └─ GET /page

next start / next dev with host span / incoming traceparent P:
 P
  └─ handleRequest GET (bubble)
      ├─ middleware GET
      └─ GET /page

separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent:
 GET (Next.request, root, inserted by the middleware adapter)
  ├─ middleware GET
  └─ GET /page

separately-invoked middleware with host span / incoming traceparent P:
 P
  ├─ middleware GET
  └─ GET /page
```

### How

- **Self-hosted:** next-server's middleware-invoke `handleRequest` pass
writes its W3C context onto the request (`traceparent`, and `tracestate`
only together with it: a received `tracestate` is removed when the
context has none). Middleware (Node in-process, or the Edge sandbox via
the header) and the render pass both continue it. The render pass forces
that extraction even under an active host span (custom server / APM), so
it nests under the bubble span; a different `traceparent` the middleware
explicitly set still wins.
- **Middleware adapter:** an active host span wins over a request
`traceparent`; otherwise the request's `traceparent` is continued, and
the middleware never changes it. Without any valid context (and OTEL
enabled) the adapter inserts a `Next.request` SERVER span (new
`NextSpan` enum) named after the method (`http.target` keeps the path).
When the downstream request doesn't already carry that shared parent,
its `traceparent`/`tracestate` are forwarded via the existing
`x-middleware-request-*` overrides, always as a pair: a user-set
`traceparent` keeps exactly the trace headers the user set. No headers
are added without OTEL.
- Request-header overrides replace the whole header set, so when the
user set no overrides, the received headers are listed alongside the
trace headers. There's no API to add a single header yet; that needs a
server and platform change.
- Node middleware is no longer detached via `ROOT_CONTEXT` (the 16.3
regression from #95306).

The inserted `Next.request` root ends when the middleware finishes, so
the render child (a separate invocation) can outlive it.

## Verification

- Trace shapes are asserted as printed span trees
(`test/lib/otel-span-tree.ts`, from #100349) in inline snapshots. Span
IDs are normalized away, but distinct traces keep distinct labels, so a
split trace shows up as `[trace A]` / `[trace B]`.
- `test/e2e/app-dir/otel-middleware-trace` (added in #100349 with the
previous behavior): this PR's snapshot diff shows middleware and render
becoming siblings in one trace, in dev, start, Cache Components and
deploy, for Node.js and Edge middleware and behind a custom server with
a host span (including an incoming `tracestate` not being paired with
the host span's context).
- Collector suite `test/e2e/opentelemetry/instrumentation`: the
middleware test asserts the full span tree in next start/dev, direct
entrypoints with/without host span, and the custom server (host span,
user-set `traceparent`).
- Unit tests for the tracer and `forwardTraceContextToRequest` changes.
- `test/e2e/middleware-request-header-overrides`,
`test/production/typescript-basic`,
`test/production/middleware-typescript` pass.

<!-- NEXT_JS_LLM -->

<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
(cherry picked from commit 4abb437)

Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
mischnic pushed a commit that referenced this pull request Oct 22, 2026
…0294, #100324, #100350, #100552, #100583) (#100542)

Backports the OpenTelemetry request-trace stack to
`releases/lts/active`:

- #100349 Start the request span before route module preparation
- #100294 Keep middleware and render in one OpenTelemetry trace as
siblings (re-lands #100136, whose LTS backport #100226 was reverted in
#100269)
- #100324 docs: note that instrumentation-time spans don't join request
traces
- #100350 Emit instrumentation lifecycle spans in the request that
registered it
- #100552 Preserve deferred middleware instrumentation spans when the
adapter throws
- #100583 test: add non-adapter deploy snapshots to
otel-middleware-trace. Without it, `otel-middleware-trace` fails the
release deploy tests that deploy with `NEXT_ENABLE_ADAPTER=0`, as on
v16.5.0-canary.23.

The six canary merge commits are cherry-picked in order with `-x`, and
all applied cleanly. The patch content is identical to canary's; only
blob IDs and line offsets differ.

## Verification

- `test/e2e/opentelemetry/instrumentation/`: `test-start-turbo` (122
passed) and `test-dev-turbo` (67 passed). The #100552 regression test
fails without its `middleware.ts` change.
- `test/e2e/app-dir/otel-middleware-trace`: `test-start-turbo`,
`test-dev-turbo` and `test-start-webpack` (30 passed each, inline
snapshots unchanged). The #100583 commit leaves the test file
byte-identical to canary's.
- Unit tests for `instrumentation-globals`, `tracer` and
`forward-trace-context` (40 passed), plus `pnpm --filter=next types`
- Built with `CI=true pnpm build-all` against the published `@next/swc`
16.4.2 binary. The backport touches no Rust.
- PR CI deploys only through the adapter, so the non-adapter snapshots
from #100583 are confirmed by the next release deploy run.

<!-- NEXT_JS_LLM -->


<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

---------

Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Oct 30, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Next.js middleware tracing produces two traces per request

3 participants