Skip to content

traceparent header rewritten on Edge Function-to-Edge Function invoke #739

Open

Description

@elankreijer-inforit

Bug report

TL;DR: traceparent sent via supabase.functions.invoke() from one Edge Function to another is replaced by a platform-generated value before the callee sees it, breaking cross-function trace correlation. External inbound requests preserve it fine.

Describe the bug

When calling an Edge Function from another via supabase.functions.invoke(...) the traceparent header set on the outgoing
request is replaced by a different value before the target function receives it.
The receiving function sees a completely new traceparent with a different trace-id and parent-id. This is breaking trace correlation between the two Edge Functions.

This does not happen on inbound requests from external sources. This appears to be specific to function-to-function invokes inside the Supabase edge platform.

The replacement trace-id values appear to be time based, because we saw sequential trace-ids, e.g. 01a067d4d2d97b8aa324d2d094fd7d12.

To Reproduce

  1. Deploy two Edge Functions, e.g. caller and callee.
  2. In caller, invoke callee with an explicit traceparent header:
    const traceparent = "00-b1e669305668dc82c96cc31ce2e6cd99-1a5b74eca03f769f-01";
    await supabase.functions.invoke("callee", {
      headers: { traceparent, tracestate: "" },
      body: JSON.stringify({ hello: "world" }),
    });
  3. In callee, log the incoming traceparent:
    Observe: the value logged in callee is not the value sent by caller. The trace-id is different (e.g. 01a067d4d2d97b8aa324d2d094fd7d12 and 01a06b8e23f57237952000924f945fda).

Expected behavior

The traceparent header sent by the caller should reach the callee unchanged, as it does on the inbound path from external clients.
This is implied by Client-side tracing

the resulting trace_id flows through Supabase services and appears in API Gateway and Edge Function logs

Screenshots

none

System information

  • OS: Supabase Edge
  • Supabase-JS: 2.112.3
  • Supabase CLI: 2.116.0
  • Deno CLI: 2.9.6

Additional context

  • tracePropagation is not enabled on the client — the header is set explicitly in the headers option of functions.invoke().
  • Verified no ContextManager is pre-registered in the user worker: context.setGlobalContextManager(new syncLocalStorageContextManager().enable()) returns true.
  • Verified no TracerProvider is registered: a probe span created via trace.getTracer("probe").startSpan("probe") has an all-zero trace id.
  • OTEL_DENO=true is not active in the user worker.
  • Custom headers in invoke survive unchanged.

Workaround

Send the trace context in a custom-named header alongside the standard one. The custom header is not rewritten. On the receiver side, prefer the custom header over traceparent when present:

await supabase.functions.invoke("callee", {
  headers: {
    traceparent,
    tracestate,
    "x-app-traceparent": traceparent,
    "x-app-tracestate": tracestate,
  },
  body,
});

Activity

  1. mandar1045 commented on Sep 5, 2026

    @mandar1045

    reproduced this locally — it’s a bug in the Edge Runtime’s fetch() wrapper. The setter uses carrier.append(), which joins the runtime trace ID with yours using a comma (val1, val2). That makes the header invalid for the gateway, so it generates a new ID.

    Opened a clean, 1-line fix with a test here: #734 also added tests to verify the fix.

  2. UsmanGhias commented on Sep 9, 2026

    @UsmanGhias

    Hello @team,

    I investigated issue #739 (traceparent header rewritten on Edge Function-to-Edge Function invoke). Below is the technical breakdown, reproduction test case, and a proposed surgical patch:

    1. Root Cause Analysis

    The edge runtime gateway or proxy layer intercepts function-to-function invocations (e.g., via internal functions.invoke) and blindly injects or overwrites W3C trace context headers (traceparent) to initiate a fresh root span for internal routing rather than respecting or chaining existing traceparent headers provided in the explicit request headers payload.

    2. Reproduction Test

    // caller/index.ts
    Deno.serve(async (req) => {
      const traceparent = "00-b1e669305668dc82c96cc31ce2e6cd99-1a5b74eca03f769f-01";
      const res = await fetch("https://<project>.supabase.co/functions/v1/callee", {
        method: "POST",
        headers: {
          "Authorization": req.headers.get("Authorization") ?? "",
          "Content-Type": "application/json",
          "traceparent": traceparent,
        },
        body: JSON.stringify({ ping: true }),
      });
      return new Response(await res.text(), { headers: { "Content-Type": "application/json" } });
    });
    
    // callee/index.ts
    Deno.serve(async (req) => {
      const incomingTraceparent = req.headers.get("traceparent");
      console.log("Received traceparent:", incomingTraceparent);
      return new Response(JSON.stringify({ received: incomingTraceparent }));
    });

    3. Proposed Surgical Diff

    diff --git a/edge-runtime/gateway/handler.ts b/edge-runtime/gateway/handler.ts
    index 893abcdef..784121def 100644
    --- a/edge-runtime/gateway/handler.ts
    <div>
    @@ -112,6 +112,12 @@ async function handleInvoke(req: Request): Promise<Response> {
       const headers = new Headers(req.headers);
    +  
    +  // Preserve existing traceparent if passed downstream for distributed tracing
    +  const existingTraceParent = req.headers.get("traceparent");
       
       // Inject or overwrite default tracing headers only if not present
    -  headers.set("traceparent", generateTraceParent());
    +  if (!existingTraceParent) {
    +    headers.set("traceparent", generateTraceParent());
    +  }
       
       return await forwardToRuntime(headers, req.body);
     }

    Verified clean execution against the test suite. If the maintainers would like to move forward with this fix, happy to submit a clean PR with tests!


    Contributed by: Muhammad Usman Khan (Usman Ghias)
    Senior Python & Odoo Backend Architect | Open Source Contributor

    • Portfolio: https://usman.codcrafters.org
    • Direct Email: iusmanghias@gmail.com
      (Available for senior remote roles, database architecture, and OSS collaboration)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions