trial: reports "All trials completed successfully" and exits 0 when safe-output results contain errors

[loganrosen]

Summary

gh aw trial prints "All trials completed successfully" and exits 0 even when the trial's own result JSON contains a non-empty errors array from rejected safe-output messages.

Environment

• gh-aw: observed on v0.84.3; code path still present on main (v0.85.1)
• OS: macOS

Expected

A trial where staged safe outputs were rejected should exit nonzero, or at minimum report a prominent partial-failure conclusion with the rejected message count.

Actual

CLI output:

Trial completed for workflow: <name>
All trials completed successfully

...while the saved trial result JSON contained:

{
  "errors": [
    "Line 1: set_issue_field requires at least one of: 'field_name', 'field_node_id' fields"
  ]
}

The underlying Actions run also concluded successfully while one staged safe output was discarded.

Code pointers (on main)

• pkg/cli/trial_helpers.go — after the per-workflow loop, the function unconditionally prints console.FormatSuccessMessage("All trials completed successfully") and returns nil. Nothing inspects the collected workflowResults for safe-output errors.
• pkg/cli/trial_types.go — WorkflowTrialResult.SafeOutputs is an opaque map[string]any, so the errors array is serialized into the result file but never examined.

Impact

Automation wrapping gh aw trial (CI gates, batch trial harnesses) treats a behaviorally incomplete trial as passing. Every consumer must independently parse the result JSON and assert errors == [], which is easy to miss and undocumented.

Suggested behavior

1. After collecting artifacts, inspect each result's safe-output errors array.
2. Return a nonzero exit code when any workflow result contains safe-output errors.
3. Surface the rejected message count (and ideally the first error message) in the CLI summary instead of the unconditional success line.
4. Include an explicit pass/fail field in the --json output so consumers do not have to infer it.