Skip to content

Codex fails to report failure if unauthorised #668

Closed
#880
Closed
Codex fails to report failure if unauthorised#668
#880
Assignees
pelikhan
Labels
automationcustom-enginetest-safe-outputs

Description

@dsyme
dsyme
opened on Sep 10, 2025

The OpenAI key is invalid we don't return an error:

https://github.com/githubnext/gh-aw/actions/runs/17622421035/job/50070992826

[2025-09-10T17:54:49] stream error: exceeded retry limit, last status: 401 Unauthorized; retrying 1/5 in 216ms…
[2025-09-10T17:54:54] stream error: exceeded retry limit, last status: 401 Unauthorized; retrying 2/5 in 414ms…
[2025-09-10T17:54:58] stream error: exceeded retry limit, last status: 401 Unauthorized; retrying 3/5 in 821ms…
[2025-09-10T17:55:03] stream error: exceeded retry limit, last status: 401 Unauthorized; retrying 4/5 in 1.611s…
[2025-09-10T17:55:08] stream error: exceeded retry limit, last status: 401 Unauthorized; retrying 5/5 in 3.039s…
[2025-09-10T17:55:15] ERROR: exceeded retry limit, last status: 401 Unauthorized

Activity

  1. github-actions commented on Sep 10, 2025

    @github-actions
    github-actionsBot
    on Sep 10, 2025 – with GitHub Actions
    Contributor

    Created related issue: #669

  2. github-actions
    added
    automation
    custom-engine
    test-safe-outputs
    on Sep 10, 2025

  3. dsyme commented on Sep 10, 2025

    @dsyme
    dsyme
    on Sep 10, 2025
    CollaboratorAuthor

    Closed by e2e test cleanup

  4. dsyme
    closed this as completedon Sep 10, 2025
  5. dsyme
    reopened this on Sep 10, 2025
  6. pelikhan
    self-assigned this
    on Sep 10, 2025
  7. Copilot
    added a commit that references this issue on Sep 10, 2025

    Add test case for issue #668: 401 Unauthorized detection in Codex err…

    e9cc12b
  8. pelikhan
    added a commit that references this issue on Sep 11, 2025

    configurable error patterns for engines (#697)

    5b17daf

  9. pelikhan commented on Sep 11, 2025

    @pelikhan
    pelikhan
    on Sep 11, 2025
    Collaborator

    Should be flagged now with #697

  10. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025

    https://github.com/webgrip/n8n-application/actions/runs/17798306042/job/50591153315#step:11:163 I'm not seeing anything actionable on my end. Another thing to maybe think about: My org has a secret called OPENAI_API_KEY_CI. If somehow the name of the secret that's used to send to OAI was configurable, that would solve a lot of useless work / secret duplication. Or maybe let a manually defined env variable overwrite the secret env var used in the step if it's set?

    It's still strange, because I'm 100% sure that the key I'm using is valid. So I'm just unsure on general on what's going on in my case.

  11. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    You could try to do

    engine:
  id: codex
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    Should work, not tested though might refuse the override

  12. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025

    I'm pretty sure that won't work, this is the code

    - name: Run Codex
  run: |
    set -o pipefail
    INSTRUCTION=$(cat /tmp/aw-prompts/prompt.txt)
    export CODEX_HOME=/tmp/mcp-config
    
    # Create log directory outside git repo
    mkdir -p /tmp/aw-logs
    
    # Run codex with log capture - pipefail ensures codex exit code is preserved
    codex exec \
      -c model=gpt-5-preview \
      --full-auto "$INSTRUCTION" 2>&1 | tee /tmp/agentic-triage.log
  env:
    GITHUB_AW_PROMPT: /tmp/aw-prompts/prompt.txt
    GITHUB_AW_SAFE_OUTPUTS: ${{ env.GITHUB_AW_SAFE_OUTPUTS }}
    GITHUB_STEP_SUMMARY: ${{ env.GITHUB_STEP_SUMMARY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

    I tried putting it in env: on the rootlevel of the frontmatter segment, which results in

    env:
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    in the .lock file, but that doesn't do anything either I'm afraid, since the env var on the step itself is getting straight from the secret.

    I'm not entirely discounting I'm just making a really stupid mistake.

    EDIT:

    I was wrong, putting

    engine:
  id: codex
  model: gpt-5-preview
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    DOES result in

    - name: Run Codex
        run: |
          set -o pipefail
          INSTRUCTION=$(cat /tmp/aw-prompts/prompt.txt)
          export CODEX_HOME=/tmp/mcp-config

          # Create log directory outside git repo
          mkdir -p /tmp/aw-logs

          # Run codex with log capture - pipefail ensures codex exit code is preserved
          codex exec \
            -c model=gpt-5-preview \
            --full-auto "$INSTRUCTION" 2>&1 | tee /tmp/agentic-triage.log
        env:
          GITHUB_AW_PROMPT: /tmp/aw-prompts/prompt.txt
          GITHUB_AW_SAFE_OUTPUTS: ${{ env.GITHUB_AW_SAFE_OUTPUTS }}
          GITHUB_STEP_SUMMARY: ${{ env.GITHUB_STEP_SUMMARY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    in the compiled version. If that's in the docs somewhere I must have missed it. My bad :)

  13. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    @dsyme did quite a bit of work with the latest codex so it might be good to upgrade (there will be some renaming breaking changes sorry).

  14. 1 remaining item

  15. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025
    - name: Run Codex
  run: |
    set -o pipefail
    INSTRUCTION=$(cat /tmp/aw-prompts/prompt.txt)
    export CODEX_HOME=/tmp/mcp-config
    
    # Create log directory outside git repo
    mkdir -p /tmp/aw-logs
    
    # where is Codex
    which codex
    
    # Check Codex version
    codex --version
    
    # Authenticate with Codex
    codex login --api-key "${{ secrets.OPENAI_API_KEY }}"
    
    # Run codex with log capture - pipefail ensures codex exit code is preserved
    codex exec -c model=gpt-5-preview --search --full-auto "$INSTRUCTION" 2>&1 | tee /tmp/agentic-triage.log
  env:
    GITHUB_AW_PROMPT: /tmp/aw-prompts/prompt.txt
    GITHUB_AW_SAFE_OUTPUTS: ${{ env.GITHUB_AW_SAFE_OUTPUTS }}
    GITHUB_STEP_SUMMARY: ${{ env.GITHUB_STEP_SUMMARY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    is what I get when I compile after running

    gh extension upgrade --all
[aw]: upgraded from v0.6.2 to v0.7.0

    codex login uses the secret. If it uses the environment variable instead, it's perfect. Right now it'll always use the secret, even though the point of the environment variable is to be able to serve exactly my use case ;)

  16. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    It's not in the docs...

    @copilot

    • add a test agentic workflow for the environment variable scenario using codex (pkg/cli/workflows)
    • update documentation for engines/claude/code about configuring/overriding secrets
  17. pelikhan
    closed this as completedin #880on Sep 17, 2025

  18. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025

    Thanks for the quick response! I didn't expect that. Just goes to show how much faster these tools let us iterate.

  19. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    Indeed login wasn't using the env var. patch coming

  20. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    Upgrade again

  21. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025

    It looks like there's a new bug. When compiling with 0.7.1, this is generated:

    - name: Run Codex
  run: |
    set -o pipefail
    INSTRUCTION=$(cat /tmp/aw-prompts/prompt.txt)
    export CODEX_HOME=/tmp/mcp-config

    # Create log directory outside git repo
    mkdir -p /tmp/aw-logs

    # where is Codex
    which codex

    # Check Codex version
    codex --version

    # Authenticate with Codex
    codex login --api-key "$OPENAI_API_KEY"

    # Run codex with log capture - pipefail ensures codex exit code is preserved
    codex exec -c model=gpt-5 --search --full-auto "$INSTRUCTION" 2>&1 | tee /tmp/agentic-triage.log
  env:
    GITHUB_AW_PROMPT: /tmp/aw-prompts/prompt.txt
    GITHUB_AW_SAFE_OUTPUTS: ${{ env.GITHUB_AW_SAFE_OUTPUTS }}
    GITHUB_STEP_SUMMARY: ${{ env.GITHUB_STEP_SUMMARY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_CI }}

    Then the following error comes up: https://github.com/webgrip/n8n-application/actions/runs/17803210894/job/50608277990#step:12:37

    2025-09-17T15:48:04.7030093Z codex-cli 0.36.0
2025-09-17T15:48:04.7395552Z Successfully logged in
2025-09-17T15:48:04.7759026Z error: unexpected argument '--search' found
2025-09-17T15:48:04.7760862Z 
2025-09-17T15:48:04.7761291Z   tip: to pass '--search' as a value, use '-- --search'
2025-09-17T15:48:04.7761678Z 
2025-09-17T15:48:04.7761988Z Usage: codex exec --config <key=value> [PROMPT]
2025-09-17T15:48:04.7762254Z 
2025-09-17T15:48:04.7762442Z For more information, try '--help'.
2025-09-17T15:48:04.7813808Z ##[error]Process completed with exit code 2.

    After I manually removed --search in the compiled .lock file, I saw a green agentic workflow for the first time :D.

  22. Ryangr0 commented on Sep 17, 2025

    @Ryangr0
    Ryangr0
    on Sep 17, 2025

    @pelikhan Just making sure you see this.

  23. pelikhan
    reopened this on Sep 17, 2025

  24. pelikhan commented on Sep 17, 2025

    @pelikhan
    pelikhan
    on Sep 17, 2025
    Collaborator

    @dsyme is --search only available in later build?

  25. dsyme commented on Sep 17, 2025

    @dsyme
    dsyme
    on Sep 17, 2025
    CollaboratorAuthor

    Ah sorry yes. Our tests in https://github.com/githubnext/gh-aw-test don't cover web-search:

    I'll fix that

  26. dsyme commented on Sep 17, 2025

    @dsyme
    dsyme
    on Sep 17, 2025
    CollaboratorAuthor

    Actually strange, I see codex 0.36.0 supporting this

    dsyme@DSYME-LAPDOG:~/gh-aw$ codex --help | grep search
      --search
          Enable web search (off by default). When enabled, the native Responses `web_search` tool is available to the
dsyme@DSYME-LAPDOG:~/gh-aw$ codex --version
codex-cli 0.36.0

  27. dsyme commented on Sep 17, 2025

    @dsyme
    dsyme
    on Sep 17, 2025
    CollaboratorAuthor

    Well it's a bit whacky but you have to put that particular parameter before "exec"

    We have considerable work to do to map the permissions model we use down to codex, which has its own sandboxing built in to the CLI tool.

  28. pelikhan
    closed this as completedon Nov 4, 2025
  29. github-actions
    added a commit that references this issue on Sep 13, 2026

    Audit-only threat spec review: 2026-09-13 daily optimizer run

    551df2b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

automationcustom-enginetest-safe-outputs

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions