Skip to content

fix: re-raise LLM guardrail execution errors instead of failed validation - #7151

Open
BetterAndBetterII wants to merge 1 commit into
crewAIInc:mainfrom
BetterAndBetterII:fix/llm-guardrail-execution-error
Open

fix: re-raise LLM guardrail execution errors instead of failed validation#7151
BetterAndBetterII wants to merge 1 commit into
crewAIInc:mainfrom
BetterAndBetterII:fix/llm-guardrail-execution-error

Conversation

@BetterAndBetterII

Copy link
Copy Markdown

Summary

LLMGuardrail.__call__ caught every exception (after HookAborted) and returned (False, "Error while validating the task output: …"). Callers treat (False, …) as “output violated the guardrail” and retry via guardrail_max_retries, stuffing the provider error into the conversation.

An infrastructure/LLM failure is not a statement about the agent’s output. This change raises a distinct GuardrailExecutionError so provider/infra errors propagate instead of being retried as validation failures. Real (False, feedback) violations still retry as before. process_guardrail still emits a completed event before re-raising, matching the HookAborted path.

Changes

  • lib/crewai/src/crewai/tasks/llm_guardrail.py — add GuardrailExecutionError; raise it from __call__ on non-HookAborted exceptions
  • lib/crewai/src/crewai/utilities/guardrail.py — catch/re-raise GuardrailExecutionError after emitting LLMGuardrailCompletedEvent
  • lib/crewai/tests/test_task_guardrails.py — TDD coverage that provider errors are not validation failures and do not consume retries

Test plan

  • uv run pytest lib/crewai/tests/test_task_guardrails.py lib/crewai/tests/hooks/test_hook_abort_propagation.py -k "guardrail" -n 0 (30 passed)

Fixes #7150

…tion

Provider/infra exceptions from LLMGuardrail were returned as
(False, error-string), so callers treated them as output violations and
retried. Raise GuardrailExecutionError so retries are reserved for real
validation failures.

Fixes crewAIInc#7150
@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 068c9666-ed05-4b3e-97a6-615778807d57

📥 Commits

Reviewing files that changed from the base of the PR and between da4daad and d600027.

📒 Files selected for processing (3)
  • lib/crewai/src/crewai/tasks/llm_guardrail.py
  • lib/crewai/src/crewai/utilities/guardrail.py
  • lib/crewai/tests/test_task_guardrails.py

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

Changes

Guardrail execution errors

Layer / File(s) Summary
Guardrail error contract
lib/crewai/src/crewai/tasks/llm_guardrail.py
Adds GuardrailExecutionError. LLMGuardrail raises it for execution failures and preserves validation results separately.
Guardrail processing and task validation
lib/crewai/src/crewai/utilities/guardrail.py, lib/crewai/tests/test_task_guardrails.py
process_guardrail emits failed completion events and re-raises execution errors. Tests cover provider failures, validation outcomes, event reporting, and the absence of task retries.

Sequence Diagram(s)

sequenceDiagram
  participant LLMGuardrail
  participant process_guardrail
  participant LLMGuardrailCompletedEvent
  participant TaskExecution
  LLMGuardrail->>process_guardrail: raise GuardrailExecutionError
  process_guardrail->>LLMGuardrailCompletedEvent: emit failed completion event
  process_guardrail-->>TaskExecution: re-raise GuardrailExecutionError
  TaskExecution-->>TaskExecution: abort without retry
Loading

Suggested reviewers: lucasgomide

Merge Risk: ⚪ Minimal · up to d6000

Guardrail provider and infrastructure failures now propagate as execution errors instead of being mistaken for output violations and consuming validation retries, while genuine violations retain their existing retry behavior. No actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the primary change: re-raising LLM guardrail execution errors instead of treating them as failed validation.
Description check ✅ Passed The description directly explains the execution-error handling change, retry behavior, event emission, and test coverage.
Linked Issues check ✅ Passed The changes address issue #7150 by separating provider and infrastructure errors from validation failures, preserving failure causes, preventing retries and conversation pollution, and retaining exist…
Out of Scope Changes check ✅ Passed The implementation and tests are limited to guardrail execution-error propagation, completion events, and retry behavior required by issue #7150.
Full details: Linked Issues check

Explanation

The changes address issue #7150 by separating provider and infrastructure errors from validation failures, preserving failure causes, preventing retries and conversation pollution, and retaining existing validation behavior.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Alkur123

Copy link
Copy Markdown

Thanks for picking this up so fast. This is the shape I had in mind when I filed #7150, and of the two options in the issue this is the one I would have gone for.

One thing I checked while writing the issue that might save a reviewer some time. There are two separate retry loops that call process_guardrail, not one:

  • Agent._process_kickoff_guardrail, which retries by calling itself again
  • Task._invoke_guardrail_function, which retries in a for loop

Neither of them wraps the process_guardrail call in a try/except, so re-raising from utilities/guardrail.py should propagate cleanly out of both and I do not think task.py needs a change for this to work. Worth someone confirming
though, because that second loop has the same behaviour the issue was about: it formats guardrail_result.error through I18N_DEFAULT.errors("validation_error") and feeds it back to agent.execute_task as context before trying again.

On the breaking change question, which I would guess is the main hesitation here: a provider outage already ended in an exception before this PR. It just got there the slow way, after three full agent re-executions, and then raised
Task failed guardrail validation after N retries, which names validation when the provider was the thing that was down. So this is not turning a returned failure into a raise. It raises sooner, with a type that says what actually
happened, and without the provider's error text landing in the agent's conversation as a user turn on the way.

I also ran your branch against the three cases from the issue, using a small script that pulls LLMGuardrail.call out of the tree with ast and calls it with a stub _validate_output, so it needs no provider key. Comparing d600027
against v1.15.1:

BEFORE (v1.15.1)
provider unavailable returned (False, 'Error while validating the task output: provider unavailable')
genuine violation returned (False, 'too long by 40 words')
passing output returned (True, "the agent's answer")

AFTER (d600027)
provider unavailable RAISED GuardrailExecutionError: provider unavailable
genuine violation returned (False, 'too long by 40 words')
passing output returned (True, "the agent's answer")

So the outage case stops returning a verdict, and both controls are untouched, which was the thing I would have worried about in a change like this. Say if the script is useful for the tests or the review and I will post it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] LLMGuardrail reports an LLM/provider error as a failed validation, and the caller retries on it

2 participants