Revisit how protos gets redacted from logs

There is some logic in internal/ateinterceptors/ateinterceptors.go to redact RPC response protos. It uses proto reflection and only redacts if it finds a field named "env".

Looking at Git history I think this was from back in the time when we had k8s secrets delivered to actors via env variables, but this is gone now as secrets don't get into the actor sandbox and are instead injected by instrumenting outbound requests.

OTOH, we have some endpoints that can return sensitive content. For example, MintActorJWTResponse has a actor_jwt field which should be redacted.

A few questions we need to answer:

Do we need to keep redacting env fields?
Is the current approach of redacting only in the interceptor ok? What if we slog a proto? That won't be redacted. Can we prevent it? Should we intercept at the slog handler level and sanitize protos? See Add log sanitization #103 (comment)
Do we need to also redact MintActorJWTResponse? Sounds like yes.

See related issue #103

Good catch, this was a relic from when we accepted SecretEnv towards the beginning of the project

Greg Castle (@destijl)

FYI Greg Castle (@destijl)

Hw about this:

Annotate all protos fields that can contain sensitive content with debug_redact.
We generalize the current function to redact using proto reflection to honor debug_redact instead of hardcoding field == 'env'.
We intercept and redact in the common slog handler instead of the grpc interceptor. This way any call to slog that passes a proto sanitizes it, not just this specific call site.

Da Huang (@git286) thoughts?

Da Huang (@git286)

I briefly talked about this with Da Huang (@git286) yesterday. We went through this whole journey with K8s, so I got Gemini to write out what it would look like to just skip to the end and learn all the lessons from K8s. Gemini content below. The only reason we've been able to avoid this in K8s/GKE is because people on my team built this awesome taint tracker: go-flow-levee. We need to run that on every PR Anna Pendleton (@annapendleton) in CI.

Direct Answers to the Questions

Do we need to keep redacting env fields?
Yes, keep it as defense-in-depth. While Substrate injects credentials for outbound requests via the egress gateway rather than container env vars, container specifications (ateletpb.WorkloadSpec.Containers[].Env) still accept environment variables. User workloads or demo apps can easily pass credentials via env.
Do we need to redact MintActorJWTResponse?
Yes, immediately. actor_jwt is an active bearer token granting actor identity. Logging it in plaintext in RPC access logs compromises the credential and allows replay.
Should we intercept at the slog.Handler level and sanitize protos via reflection?
Strong recommendation: No.
Kubernetes attempted runtime dynamic log reflection in klog under KEP-1753 (k8s.io/component-base/logs/sanitization). It had to be completely removed in K8s 1.24 (PR #107207) because traversing arbitrary object trees with reflection on every log call introduced severe CPU overhead, memory allocations on hot paths, and risks of blocking the logging pipeline. Running reflection across all slog calls in high-throughput components like atelet and ateom will cause latency regressions.

The Kubernetes Solution: KEP-1933 & `go-flow-levee`

Instead of runtime reflection in the logger, Kubernetes shifted left to static taint analysis at compile/test time:

Tool: google/go-flow-levee (open-sourced by Google/GKE Security).
How it works: Uses SSA (Static Single Assignment) to track data flow from Sources (fields matching regexes like Token|Secret|Password|ActorJwt or tagged with levee:"source") through assignments/wrappers to Sinks (functions like log/slog.* or fmt.Print*). If tainted data can reach a logging sink without passing through a whitelisted Sanitizer, CI fails.
CI Integration: Kubernetes runs this via hack/verify-govet-levee.sh as a presubmit job. It catches unsanitized logs before code merges with zero runtime overhead.

Recommended Strategy for Substrate

A clean 3-layer approach that fits our current slog and serverboot setup:

1. Immediate Fix: Generalize `ateinterceptors.go` for RPC logs

In internal/ateinterceptors/ateinterceptors.go, replace the strict fd.Name() == "env" check with a set of sensitive field names:

var sensitiveFields = map[protoreflect.Name]bool{
	"env":       true,
	"actor_jwt": true,
	"token":     true,
	"secret":    true,
}

func clearSensitiveFields(msg protoreflect.Message) {
	msg.Range(func(fd protoreflect.FieldDescriptor, value protoreflect.Value) bool {
		if sensitiveFields[fd.Name()] {
			msg.Clear(fd)
			return true
		}
		if fd.IsList() {
			list := value.List()
			for i := 0; i < list.Len(); i++ {
				if fd.Kind() == protoreflect.MessageKind || fd.Kind() == protoreflect.GroupKind {
					clearSensitiveFields(list.Get(i).Message())
				}
			}
			return true
		}
		if fd.Kind() == protoreflect.MessageKind || fd.Kind() == protoreflect.GroupKind {
			clearSensitiveFields(value.Message())
		}
		return true
	})
}

2. Direct `slog` Safety: Implement `slog.LogValuer` on Sensitive Protos

Go 1.21+ slog has built-in zero-reflection masking via slog.LogValuer. In a non-generated file (e.g., pkg/proto/ateapipb/redact.go):

package ateapipb

import "log/slog"

func (x *MintActorJWTResponse) LogValue() slog.Value {
	return slog.GroupValue(
		slog.String("actor_jwt", "[REDACTED]"),
	)
}

If any internal service calls slog.Info("minted", "resp", resp), slog natively invokes LogValue() without needing custom reflection in the handler.

3. String Regex Redaction in `contextlogging.ContextHandler` (Targeted)

In internal/contextlogging/contextlogging.go (which already decorates slog.Handler across all binaries via serverboot.InitLogger()), we can optionally add regex checks for JWT (^[a-zA-Z0-9+/=_-]{5,}\.[a-zA-Z0-9+/=_-]{4,}\.[a-zA-Z0-9+/=_-]{4,}$) and PEM private keys. Critically: run this only on rec.Message and top-level string attributes, avoiding recursive struct reflection.

4. Shift Left: Add `hack/verify/levee.sh` to CI

Add google/go-flow-levee to our verification suite under hack/verify/levee.sh (executed automatically by hack/verify-all.sh in PR workflow) to statically prevent sensitive proto fields and credential types from reaching slog calls.

Da Huang (@git286)

Da Huang (@git286) can you find someone to take this on?

Sure, agreed this is a P0 for GA and the o11y team will take this.

Should we intercept at the slog.Handler level and sanitize protos via reflection?
Strong recommendation: No.

I'm not as familiar with the latest protoreflect, but IIRC there were a lot of optimizations done there, would it have the same performance implications?

Steven Shriver (@stshrive)

+cc Steven Shriver (@stshrive) FYI for OSS CI side - sounds like Da Huang (@git286) and o11y team will take this work on, I'm assuming that includes "4. Shift Left: Add hack/verify/levee.sh to CI"

To be clear that very emphatic no on question 3 came from Gemini, not me :) The security deliverable is just effective redacting, I'll defer implementation choices to people who know golang much better than me. If you looked at the k8s problems and think they don't apply, feel free to overrule Gemini.

Makes sense, I just really like having these things colocated when possible, so could be worth a quick investigation. Agents can do the comparison benchmarking quickly and usually fairly well.

Anna Pendleton (@annapendleton)

I did some digging on how k8s handles this. Short summary below, I will write a Google doc with the details and links and share it here.

How k8s does it today

k8s tried two ways and removed both. Today k8s has no automatic log sanitization.

Runtime filter (KEP-1753): tag secret fields with a Go struct tag, a klog hook walks every log argument with Go reflect and drops the whole line if it finds one. Removed in v1.24 (Deprecate dynamic log sanitization kubernetes/kubernetes#107207), too slow, missed pointer fields, and never on by default.
Static analysis with go-flow-levee (KEP-1933): presubmit that fails the PR if a secret can reach a klog call. Removed in Jan 2024 (Drop hack/verify-govet-levee kubernetes/kubernetes#122596), the tool is abandoned and does not support generics. I tried it on our repo, it panics before reaching our code, so we cannot use it in CI.

What k8s relies on now is a few hand-written String() methods that redact, an audit policy that never captures Secret bodies, and code review. The field tags are still there but nothing reads them. k8s also added debug_redact to the CRI API in v1.34 (kubernetes/kubernetes#133135), also with no reader yet.

So the one idea that survived is the simplest: label the field.

On the perf concern

The k8s filter walked every value of every log call with Go reflect. We only touch proto messages, can use protoreflect, and can cache per message type whether it has any sensitive field, so most messages cost nothing. We already do this walk in the interceptor for every RPC today. We will benchmark before we commit to the handler approach.

Plan (high level)

Label sensitive proto fields with debug_redact, make the redaction honor the label instead of hardcoded names, and fix the known leaks first.
Benchmark interceptor vs shared slog handler, and move redaction into the shared handler if the numbers are fine.
Static check: we will look into it separately, not part of the P0. Levee is not an option, it would need to be a small custom check.

Anna Pendleton (@annapendleton) Steven Shriver (@stshrive) item 4 (levee in CI) is not something we can deliver, see above and let me know if I miss anything.

ateinterceptors: redact debug_redact fields from RPC logs (agent-subs…

contextlogging: redact protobuf attributes in the shared slog handler (…

docs: add debug_redact guidance to the API and code style guides

docs: add debug_redact guidance to the API style guide and a logging …

Da Huang (@git286)

The M3 deliverable is that we're not leaking tokens, a more sophisticated leakproofing strategy can come after M3. Da Huang (@git286) I believe we are at that state already after your changes, so I am removing M3 from this issue. Please add it back if there's still stopleaks work to do.

docs: add debug_redact guidance to the API style guide and a logging …

docs: say when to mark proto fields debug_redact and what it covers (a…

This is implemented now. Closing.

Navigation Menu

Repository navigation

Revisit how protos gets redacted from logs #1743

Direct Answers to the Questions

The Kubernetes Solution: KEP-1933 & `go-flow-levee`

Recommended Strategy for Substrate

1. Immediate Fix: Generalize `ateinterceptors.go` for RPC logs

2. Direct `slog` Safety: Implement `slog.LogValuer` on Sensitive Protos

3. String Regex Redaction in `contextlogging.ContextHandler` (Targeted)

4. Shift Left: Add `hack/verify/levee.sh` to CI

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Revisit how protos gets redacted from logs #1743

Description

Activity

EItanya commented on Sep 18, 2026

juli4n commented on Sep 18, 2026

destijl commented on Sep 18, 2026

Direct Answers to the Questions

The Kubernetes Solution: KEP-1933 & go-flow-levee

Recommended Strategy for Substrate

1. Immediate Fix: Generalize ateinterceptors.go for RPC logs

2. Direct slog Safety: Implement slog.LogValuer on Sensitive Protos

3. String Regex Redaction in contextlogging.ContextHandler (Targeted)

4. Shift Left: Add hack/verify/levee.sh to CI

destijl commented on Sep 18, 2026

git286 commented on Sep 18, 2026

EItanya commented on Sep 18, 2026

annapendleton commented on Sep 18, 2026

destijl commented on Sep 18, 2026

EItanya commented on Sep 18, 2026

git286 commented on Sep 21, 2026

8 remaining items

destijl commented on Oct 13, 2026

juli4n commented on Oct 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The Kubernetes Solution: KEP-1933 & `go-flow-levee`

1. Immediate Fix: Generalize `ateinterceptors.go` for RPC logs

2. Direct `slog` Safety: Implement `slog.LogValuer` on Sensitive Protos

3. String Regex Redaction in `contextlogging.ContextHandler` (Targeted)

4. Shift Left: Add `hack/verify/levee.sh` to CI