Skip to content

fix(flags): warn once when the secret key can't read flag definitions - #240

Closed
Silthus wants to merge 1 commit into
mainfrom
fix/flag-definitions-scope-warning
Closed

Silthus wants to merge 1 commit into
mainfrom
fix/flag-definitions-scope-warning

Conversation

@Silthus

@Silthus Silthus commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

When you configure a secret_key, local evaluation is on by default and PostHog.FeatureFlags.DefinitionLoader polls /flags/definitions with it. Transactional email asks for a Project Secret API Key with only the transactional_email:write scope. The flags service answers that key with 401 authentication_failed ("Secret API token is invalid."), the same answer as a wrong key, because its key lookup only finds keys with a flag scope. A personal API key without the scope gets 403 permission_denied.

So an app that only sends email logged definition refresh was rejected with HTTP 401 on every poll, for as long as it ran, about feature flags it never uses.

Merge Danger

Low, two-way door. Only the logging for 401 and 403 from /flags/definitions changes. Definitions still clear on 401 and 403, and polling, 402, 429, timeouts, provider paths and remote evaluation are unchanged.

What changes

  • On 401 or 403, the loader logs one warning. It says the key is invalid or lacks the Feature flags: Read scope, and that enable_local_evaluation: false turns local evaluation off.
  • Later rejected polls log at debug, with posthog_skip_capture: true like the other loader logs. Any successful refresh (a 200 install, a 304 or a cache install) re-arms the warning, so a key that breaks later warns again.
  • The loader keeps its poll interval. A user who adds the scope gets local evaluation back on the next successful poll, and after a restart.
  • 402 keeps its message and its warning on every poll.

This PR is independent of the email PR #236. That PR doesn't touch the definition loader or its tests, and a merge of both is clean.

💚 How did you test it?

  • New tests in test/posthog/feature_flags/definition_loader_test.exs: a start plus four refreshes answered with 401 or 403 log one warning (none matched before the fix, which logged the old message every time), and a 200 between rejections warns again. They failed before the fix and pass now.
  • mix test: 869 passed, 18 excluded. mix format --check-formatted, mix credo --strict, mix compile --warnings-as-errors and mix posthog.public_api --check pass.
  • Test seam: the public PostHog.Supervisor with PostHog.API.Mock answering /flags/definitions, read through capture_log.
  • Live run: PostHog.Supervisor with secret_key and feature_flags_poll_interval_ms: 1_000 polled a local stub of /flags/definitions that sends the flags service's 401 and 403 bodies. Over 8 polls (three 401s, a 200, then 401s) it logged two warnings, one before and one after the load, and debug lines for the rest. Five 403s logged one warning.
  • Gap: no run against PostHog itself with a scope-less key. The stub's 401 and 403 come from reading rust/feature-flags/src/api/auth.rs and errors.rs in PostHog/posthog.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Claude Code (Claude Opus 5.5) wrote the change test-first in a T3 Code thread, as a side fix from the transactional email work. GPT-6.1 Sol reviewed the diff before the push and found nothing that holds. The same fix is going to posthog-node, posthog-python and posthog-go.

Question Answer Who Where
Which responses count as a key that can't read flags? 401 and 403, because the flags service answers a scope-less project secret key with the same 401 as a wrong key 🤖 Agent comment
Stop polling or keep polling quietly? Keep polling at the configured interval and stay quiet, so a transient 401 or an added scope recovers without a restart 🤖 Agent comment
Which channel reports it? One Logger.warning, then Logger.debug 🤖 Agent comment

Involvement: one cross-family review (GPT-6.1 Sol), no grilling rounds, live-run proof against a local stub of the flags service.

🤖 Generated with Claude Code

A secret key without the Feature flags: Read scope gets 401 from
/flags/definitions on every poll, so the definition loader logged a
warning every poll. Warn once for 401 and 403 with the scope and the
enable_local_evaluation: false opt-out, and again only after definitions
load. Definitions still clear, and 402 keeps its per-poll warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus Silthus self-assigned this Nov 5, 2026
@Silthus

Silthus commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decisions made while building this fix:

  • 🤖 Which responses count as a key that can't read flags? 401 and 403. The flags service answers a project secret key without the flag scope with the same 401 as a wrong key, so the SDK can't tell them apart and the message names both causes.
  • 🤖 Stop polling or keep polling quietly? Keep polling at the existing cadence and log nothing more until definitions load. Stopping would turn a transient 401 into a dead poller until restart.
  • 🤖 Which channel reports it? The SDK's existing warning or error channel, once. Other statuses keep their current logging.

@posthog posthog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed d6a1878: no findings.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

posthog-elixir Compliance Report

Date: 2026-11-05T12:15:00.092207+00:00
Duration: 108650ms

⚠️ Some Tests Failed

40/47 tests passed, 7 failed


Capture Tests

⚠️ 29/30 tests passed, 1 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 610ms
Format Validation.Event Has Uuid ✅ 610ms
Format Validation.Event Has Lib Properties ✅ 610ms
Format Validation.Distinct Id Is String ✅ 609ms
Format Validation.Token Is Present ✅ 610ms
Format Validation.Custom Properties Preserved ✅ 610ms
Format Validation.Event Has Timestamp ✅ 609ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ❌ 610ms
Retry Behavior.Retries On 503 ✅ 5615ms
Retry Behavior.Does Not Retry On 400 ✅ 2613ms
Retry Behavior.Does Not Retry On 401 ✅ 2613ms
Retry Behavior.Respects Retry After Header ✅ 5614ms
Retry Behavior.Implements Backoff ✅ 15624ms
Retry Behavior.Retries On 500 ✅ 5616ms
Retry Behavior.Retries On 502 ✅ 5616ms
Retry Behavior.Retries On 504 ✅ 5616ms
Retry Behavior.Max Retries Respected ✅ 15618ms
Deduplication.Generates Unique Uuids ✅ 622ms
Deduplication.Preserves Uuid On Retry ✅ 5614ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10615ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5618ms
Deduplication.No Duplicate Events In Batch ✅ 616ms
Deduplication.Different Events Have Different Uuids ✅ 615ms
Compression.Sends Gzip When Enabled ✅ 610ms
Batch Format.Uses Proper Batch Structure ✅ 610ms
Batch Format.Flush With No Events Sends Nothing ✅ 607ms
Batch Format.Multiple Events Batched Together ✅ 615ms
Error Handling.Does Not Retry On 403 ✅ 2612ms
Error Handling.Does Not Retry On 413 ✅ 2613ms
Error Handling.Retries On 408 ✅ 5616ms

Failures

format_validation.non_utc_event_timestamp_is_converted_to_utc

Event 0 field 'timestamp' instant '2026-11-05T12:13:16.023740Z' != expected '2025-01-02T03:04:05Z'

Feature_Flags Tests

⚠️ 11/17 tests passed, 6 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 9ms
Request Payload.Flags Request Uses V2 Query Param ✅ 7ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 7ms
Request Payload.Flags Request Omits Authorization Header ✅ 7ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 7ms
Request Payload.Groups Default To Empty Object ❌ 7ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 7ms
Request Payload.Disable Geoip Omitted Defaults To False ❌ 7ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 7ms
Request Lifecycle.No Flags Request On Init Alone ✅ 4ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 609ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 11ms
Request Lifecycle.Mock Response Value Is Returned To Caller ❌ 7ms
Retry Behavior.Retries Flags On 502 ❌ 310ms
Retry Behavior.Retries Flags On 504 ❌ 311ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ❌ 611ms

Failures

request_payload.groups_default_to_empty_object

Field 'groups' not found in /flags request body at path 'groups'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']

request_lifecycle.mock_response_value_is_returned_to_caller

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_502

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_504

Last action result missing field 'value'. Keys: ['error', 'success']

side_effect_events.get_feature_flag_captures_feature_flag_called_event

Expected 1 events with name '$feature_flag_called', got 0

@github-actions

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

@github-actions github-actions Bot added the stale label Nov 13, 2026
@Silthus Silthus closed this Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant