fix!: classify severity with a default rubric and configurable reviews

[kmbroai]

Summary

Severity classification currently uses one model answer with a custom rubric and otherwise returns the scanner's original severity. Classify uncached findings with a bundled rubric by default and combine three independent reviews. Allow callers to request 1–10 reviews, returning the most common severity and choosing the highest severity among tied top counts.

Changes

• Bundle a readable Critical/High/Medium/Low/Informational policy based on attacker access, boundary crossing, incremental harm, and evidence. A custom rubric replaces it completely.
• Apply independent reviews consistently to saved/imported findings and canonical records. Each local review starts a fresh Codex thread; canonical reviews have unique host request IDs and identical immutable inputs. Canonical inputs may omit the rubric and receive the bundled policy's revision/hash binding.
• Add --runs N to classify-severity, including --records, with integer bounds 1–10 and default 3. SDK classification options use classificationRuns. Reject invalid counts before model work. One review returns its assessment directly.
• Vote by normalized severity, retain the first winning answer's rationale and metadata, and use severity ordering to resolve ties among the most frequent levels. Exclusion is a distinct vote and loses ties to any assessed severity.
• Require every requested review to succeed. Preserve failure/cancellation behavior, checkpoints, host accounting, and existing authorization; never use a partial vote or fall back to scanner severity.
• Record classificationRuns in results and saved assessment provenance. Append a database migration and invalidate caches when the count changes or was not recorded. Preserve read-only access to older databases and existing results; changing default/custom policy also invalidates matching checkpoints.
• Update help, documentation, package validation, and focused regression coverage.

Testing

• 114 focused SDK tests passed, covering all 216 default vote combinations, configurable bounds/counts, even ties, the tenth review failing, local/canonical execution, checkpoint reuse, and old database reads. The changed-count partial-failure/cache-resumption regression also passed.
• Four native workbench tests and 130 publication/packaging regression tests passed. The Python migration suite passed 109 tests (one skipped), after updating its expected migration history for migration 51.
• Types, production/CI builds, formatter/Markdown checks, plugin bundle build, portable source checks, and git diff --check passed.
• Full installed-package smoke passed: public imports, NodeNext types, CLI/SDK lifecycle, hosted protocol, bundled workers, and detached restart/finalization.
• Independent review found no actionable issues; reviewer separately passed all 114 focused SDK tests and four native workbench tests.
• Broader plugin suite: 426 passed, five skipped, three failed. All three failures reproduce on an untouched baseline checkout (a file-watcher limit, a raw-byte path permission error, and an unexpected trace file).
• Full seeded SDK run: 5,227 passed, 78 skipped, 22 failed. All 22 distinct failure names match the prior baseline reproductions (export/helper path aliases, publication, hook installation, and release-workflow fixtures). Three tests already proven incompatible with the local sandbox were excluded by name; no test source was disabled.
• Final-head CI passed on e0f31a68. The unchanged macOS native pipe-error assertion initially received ENOTCONN instead of EPIPE; its shard passed one retry without code changes. All other checks passed, and the automated security review completed with no findings.

Risk and rollout

Default classification now requires model access instead of inheriting scanner severity at no cost. Each uncached finding uses the requested number of sequential model calls (three by default). Existing per-turn Codex usage and host-owned accounting remain in place, including calls before a later failure; no token rollup is added. Old caches without recorded review counts are reclassified on the next classification request. Use --reprocess when changing only model/effort. Result versions and permissions are unchanged; classificationRuns is additive provenance. No merge, release, deployment, or feature-flag change is included.

Tests verify execution and voting mechanics. No model accuracy or severity-calibration evaluation has been run; the highest-severity tie-break still warrants an overclassification evaluation.

Public disclosure review

• No customer, partner, prospect, or user identities, data, or identifying details are included.
• No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
• I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	🔄 Running since 2026-10-20T07:59:37.691278Z	a31980c	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[mldangelo-oai]

non blocking comment. Would be great if there were some evals to justify this change. Sometimes features like this seem helpful at first but are harmful to the product. I am especially thinking about all of the feedback we get about classifying severities as too high.

[daneschneider-oai]

The rubric text lgtm!

[kmbroai]

non blocking comment. Would be great if there were some evals to justify this change. Sometimes features like this seem helpful at first but are harmful to the product. I am especially thinking about all of the feedback we get about classifying severities as too high.

@mldangelo-oai
This is not actually changing anything that customers use today. it is additive