Repository navigation
Keep requests through middleware in a single OpenTelemetry trace - #100136
Merged
Merged
Conversation
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Contributor
Tests PassedCommit: 5d1fa1a |
…xture Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
sokra
commented
Oct 14, 2026
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com>
Member
Author
|
Reviewed 8ef9202 |
sokra
marked this pull request as ready for review
October 14, 2026 11:00
lukesandberg
approved these changes
Oct 14, 2026
Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
This was referenced Oct 16, 2026
mischnic
approved these changes
Oct 16, 2026
sokra
added a commit
that referenced
this pull request
Oct 16, 2026
sokra
added a commit
that referenced
this pull request
Oct 16, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 17, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 19, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic
added a commit
that referenced
this pull request
Oct 19, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
sokra
added a commit
that referenced
this pull request
Oct 21, 2026
…race" (#100136) This reverts commit f1a69d7 (#100302). Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic
added a commit
that referenced
this pull request
Oct 21, 2026
…00294) ## Summary Stacked on #100349 (request span before route module preparation). Relands #100136 (reverted in f1a69d7) and changes the span shape so middleware and the route are **siblings** in one trace per request. ### Resulting shapes ``` next start / next dev: handleRequest GET (bubble, root) ├─ middleware GET └─ GET /page next start / next dev with host span / incoming traceparent P: P └─ handleRequest GET (bubble) ├─ middleware GET └─ GET /page separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent: GET (Next.request, root, inserted by the middleware adapter) ├─ middleware GET └─ GET /page separately-invoked middleware with host span / incoming traceparent P: P ├─ middleware GET └─ GET /page ``` ### How - **Self-hosted:** next-server's middleware-invoke `handleRequest` pass writes its W3C context onto the request (`traceparent`, and `tracestate` only together with it: a received `tracestate` is removed when the context has none). Middleware (Node in-process, or the Edge sandbox via the header) and the render pass both continue it. The render pass forces that extraction even under an active host span (custom server / APM), so it nests under the bubble span; a different `traceparent` the middleware explicitly set still wins. - **Middleware adapter:** an active host span wins over a request `traceparent`; otherwise the request's `traceparent` is continued, and the middleware never changes it. Without any valid context (and OTEL enabled) the adapter inserts a `Next.request` SERVER span (new `NextSpan` enum) named after the method (`http.target` keeps the path). When the downstream request doesn't already carry that shared parent, its `traceparent`/`tracestate` are forwarded via the existing `x-middleware-request-*` overrides, always as a pair: a user-set `traceparent` keeps exactly the trace headers the user set. No headers are added without OTEL. - Request-header overrides replace the whole header set, so when the user set no overrides, the received headers are listed alongside the trace headers. There's no API to add a single header yet; that needs a server and platform change. - Node middleware is no longer detached via `ROOT_CONTEXT` (the 16.3 regression from #95306). The inserted `Next.request` root ends when the middleware finishes, so the render child (a separate invocation) can outlive it. ## Verification - Trace shapes are asserted as printed span trees (`test/lib/otel-span-tree.ts`, from #100349) in inline snapshots. Span IDs are normalized away, but distinct traces keep distinct labels, so a split trace shows up as `[trace A]` / `[trace B]`. - `test/e2e/app-dir/otel-middleware-trace` (added in #100349 with the previous behavior): this PR's snapshot diff shows middleware and render becoming siblings in one trace, in dev, start, Cache Components and deploy, for Node.js and Edge middleware and behind a custom server with a host span (including an incoming `tracestate` not being paired with the host span's context). - Collector suite `test/e2e/opentelemetry/instrumentation`: the middleware test asserts the full span tree in next start/dev, direct entrypoints with/without host span, and the custom server (host span, user-set `traceparent`). - Unit tests for the tracer and `forwardTraceContextToRequest` changes. - `test/e2e/middleware-request-header-overrides`, `test/production/typescript-basic`, `test/production/middleware-typescript` pass. <!-- NEXT_JS_LLM --> <!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
mischnic
pushed a commit
that referenced
this pull request
Oct 21, 2026
…00294) ## Summary Stacked on #100349 (request span before route module preparation). Relands #100136 (reverted in f1a69d7) and changes the span shape so middleware and the route are **siblings** in one trace per request. ### Resulting shapes ``` next start / next dev: handleRequest GET (bubble, root) ├─ middleware GET └─ GET /page next start / next dev with host span / incoming traceparent P: P └─ handleRequest GET (bubble) ├─ middleware GET └─ GET /page separately-invoked middleware (Vercel, adapters, direct entrypoints), no parent: GET (Next.request, root, inserted by the middleware adapter) ├─ middleware GET └─ GET /page separately-invoked middleware with host span / incoming traceparent P: P ├─ middleware GET └─ GET /page ``` ### How - **Self-hosted:** next-server's middleware-invoke `handleRequest` pass writes its W3C context onto the request (`traceparent`, and `tracestate` only together with it: a received `tracestate` is removed when the context has none). Middleware (Node in-process, or the Edge sandbox via the header) and the render pass both continue it. The render pass forces that extraction even under an active host span (custom server / APM), so it nests under the bubble span; a different `traceparent` the middleware explicitly set still wins. - **Middleware adapter:** an active host span wins over a request `traceparent`; otherwise the request's `traceparent` is continued, and the middleware never changes it. Without any valid context (and OTEL enabled) the adapter inserts a `Next.request` SERVER span (new `NextSpan` enum) named after the method (`http.target` keeps the path). When the downstream request doesn't already carry that shared parent, its `traceparent`/`tracestate` are forwarded via the existing `x-middleware-request-*` overrides, always as a pair: a user-set `traceparent` keeps exactly the trace headers the user set. No headers are added without OTEL. - Request-header overrides replace the whole header set, so when the user set no overrides, the received headers are listed alongside the trace headers. There's no API to add a single header yet; that needs a server and platform change. - Node middleware is no longer detached via `ROOT_CONTEXT` (the 16.3 regression from #95306). The inserted `Next.request` root ends when the middleware finishes, so the render child (a separate invocation) can outlive it. ## Verification - Trace shapes are asserted as printed span trees (`test/lib/otel-span-tree.ts`, from #100349) in inline snapshots. Span IDs are normalized away, but distinct traces keep distinct labels, so a split trace shows up as `[trace A]` / `[trace B]`. - `test/e2e/app-dir/otel-middleware-trace` (added in #100349 with the previous behavior): this PR's snapshot diff shows middleware and render becoming siblings in one trace, in dev, start, Cache Components and deploy, for Node.js and Edge middleware and behind a custom server with a host span (including an incoming `tracestate` not being paired with the host span's context). - Collector suite `test/e2e/opentelemetry/instrumentation`: the middleware test asserts the full span tree in next start/dev, direct entrypoints with/without host span, and the custom server (host span, user-set `traceparent`). - Unit tests for the tracer and `forwardTraceContextToRequest` changes. - `test/e2e/middleware-request-header-overrides`, `test/production/typescript-basic`, `test/production/middleware-typescript` pass. <!-- NEXT_JS_LLM --> <!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Tobias Koppers <1365881+sokra@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com> Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com> (cherry picked from commit 4abb437) Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
mischnic
pushed a commit
that referenced
this pull request
Oct 22, 2026
…0294, #100324, #100350, #100552, #100583) (#100542) Backports the OpenTelemetry request-trace stack to `releases/lts/active`: - #100349 Start the request span before route module preparation - #100294 Keep middleware and render in one OpenTelemetry trace as siblings (re-lands #100136, whose LTS backport #100226 was reverted in #100269) - #100324 docs: note that instrumentation-time spans don't join request traces - #100350 Emit instrumentation lifecycle spans in the request that registered it - #100552 Preserve deferred middleware instrumentation spans when the adapter throws - #100583 test: add non-adapter deploy snapshots to otel-middleware-trace. Without it, `otel-middleware-trace` fails the release deploy tests that deploy with `NEXT_ENABLE_ADAPTER=0`, as on v16.5.0-canary.23. The six canary merge commits are cherry-picked in order with `-x`, and all applied cleanly. The patch content is identical to canary's; only blob IDs and line offsets differ. ## Verification - `test/e2e/opentelemetry/instrumentation/`: `test-start-turbo` (122 passed) and `test-dev-turbo` (67 passed). The #100552 regression test fails without its `middleware.ts` change. - `test/e2e/app-dir/otel-middleware-trace`: `test-start-turbo`, `test-dev-turbo` and `test-start-webpack` (30 passed each, inline snapshots unchanged). The #100583 commit leaves the test file byte-identical to canary's. - Unit tests for `instrumentation-globals`, `tracer` and `forward-trace-context` (40 passed), plus `pnpm --filter=next types` - Built with `CI=true pnpm build-all` against the published `@next/swc` 16.4.2 binary. The backport touches no Rust. - PR CI deploys only through the adapter, so the non-adapter snapshots from #100583 are confirmed by the next release deploy run. <!-- NEXT_JS_LLM --> <!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e --> --------- Co-authored-by: vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by: Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Requests that pass through middleware /
proxy.tsnow produce one OpenTelemetry trace, instead of two or three.Before, with OTEL configured in
instrumentation.ts:next()/ rewrite)After: 1 trace in every case.
Why?
Two independent causes:
BaseServer.handleRequesttwice: a middleware-invoke pass, then a render pass. Each pass extracts its context from the incoming request headers. Without an incomingtraceparent, each pass started its own root trace.ROOT_CONTEXTto makeMiddleware.executea sibling ofhandleRequest. That also detached it from any outer span, so the middleware span always started an extra trace.Customers upgrading 16.2 → 16.3 see this as "doubled traces".
How?
When middleware lets a request continue, the render is always nested inside the middleware span, in every host:
Middleware → render. The middleware adapter forwards the
Middleware.executespan's own W3C context (traceparent/tracestate, via the configured propagator) as request-header overrides onnext()and rewrite responses. Every host already applies these overrides to the downstream request, including thenext startrouter server, so no new host protocol is needed.traceparent/tracestatethe middleware explicitly set (different from the one it received) wins. A value copied unchanged from the request, as withnew Headers(request.headers), is replaced so the render still nests.Request → middleware (self-hosted). Once the middleware-invoke pass's
handleRequestspan is active, its W3C context is written onto the request headers. The middleware adapter (Node.js and the Edge sandbox) extracts it, so the middleware span is a child of that pass. The middleware-pass span keeps its existing name (GET); the route stays on the render-pass span. If there is no middleware span to forward (for example OTEL is registered only for Node.js while middleware runs on Edge), the render falls back to this context and still stays in one trace.#95306 regression. The
ROOT_CONTEXTdetachment is removed, so an outer span, an incomingtraceparent, or the middleware-pass span now parents Node.js middleware normally.All of this only runs when OpenTelemetry is active and a valid span context exists. Without OTEL, no headers are added.
One-time initialization traces stay separate by design: instrumentation registration, route-module prewarming, and the once-per-process lazy
RouteModule.prepareof direct entrypoints.Tests
test/e2e/app-dir/otel-middleware-single-trace. It runs Node.jsproxy.tsand Edgemiddleware.tswith@vercel/otel. Middleware reports its active trace and span IDs; the render reports its active trace ID and thetraceparentit received. The test asserts one trace and that the render's parent is the middleware span, so no collector is needed. It runs in dev, start and deploy mode, with no deploy gate. It covers:traceparent;traceparent;x-middleware-*headers reaching the client.test/e2e/opentelemetry/instrumentation. The middleware tests now assert real trace IDs and the exact shape: the render span's parent is the middleware span, the middleware span's parent is the middleware-passhandleRequest(self-hosted) or the external / host parent. They also check every span exported for the request.traceparent.forward-trace-context.test.ts), including unchanged copied trace headers.Verification
handleRequestor the external span) and pass after it.__NEXT_CACHE_COMPONENTS=true.test/e2e/opentelemetry/instrumentation/passes in start mode (Turbopack); the middleware tests also pass in dev (Turbopack) and start (webpack).test/e2e/opentelemetry/client-trace-metadata,test/e2e/app-dir/otel-parent-span-propagation,test/e2e/middleware-request-header-overrides, and the W3C trace-header fetch-cache test intest/e2e/app-dir/app-static.Fixes #91282