Skip to content

test_key_is_never_forwarded fails locally for any developer with GFLOW_CLI_LLM_API_KEY in .env #758

Closed

Description

@ffroliva

Found during the v0.71.1 release gate (step 4 of /gflow:release). Not a release blocker — CI is green on the identical code — but it is a real isolation gap in a test that guards a security property.

Symptom

tests/test_config.py::TestRemovedGeminiKeyNotice::test_key_is_never_forwarded
E   AssertionError: assert SecretStr('**********') is None

1 failed, 4086 passed. Reproduces in isolation in 3.8 s.

Cause

monkeypatch.setenv("GFLOW_CLI_GEMINI_API_KEY", "AIza-old")
monkeypatch.delenv("GFLOW_CLI_LLM_API_KEY", raising=False)
assert Settings().llm_api_key is None

delenv clears the process environment variable. Settings reads two sources — the env var and the .env files returned by config.py::_env_files(). So on any machine whose .env sets GFLOW_CLI_LLM_API_KEY, pydantic-settings loads it back from the file and the assertion fails.

CI has no .env, so it is green there — PR #757's matrix passed on 3.11/3.12/3.13 with this exact code.

Why it is worth fixing rather than tolerating

The test exists to prove a removed key is never resurrected as a fallback. It fails only for the developers who actually have a key configured — which is precisely the population whose environment could expose the bug it guards against. A red that everyone learns to ignore locally is worse than no test, and this is the second .env-vs-env-var isolation issue in this area (cf. the .env.local correction in AGENTS.md).

Suggested fix

Neutralise the file source for this test, not just the process env. Either point _env_files() at the tmp path the fixture already creates, or assert against a Settings built with _env_file=None. The autouse _isolate_settings fixture in tests/conftest.py already redirects GFLOW_CLI_HOME, so the home-side .env is covered — it is the CWD-relative .env that leaks.

A tests/ sweep for other delenv("GFLOW_CLI_*") uses would be worth doing in the same pass; any of them that then constructs Settings() has the same hole.

Not verified

Whether other tests share the defect — I only ran the one that failed. The full offline suite is otherwise green (4086 passed, 24 skipped).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions