[Richard Lundeen (richlundeen)]

Description

Refresh the registered privacy harm scorer's legacy metrics with explicit input coverage and a constant-guess baseline, and record a complete Azure GPT-4o temperature-zero configuration comparison. This is measurement completeness and useful negative evidence, not an accuracy improvement or a scorer promotion.

Privacy was selected because its existing mean absolute error (MAE) was worse than a constant-median guess on its own gold set. The objective/default refresh in #4234 and task-achieved retirement in #4249 are already covered by open PRs and are not repeated here.

• Replace the existing likert_privacy_gpt4o measurement for the same eval hash, 11412a611f8c19b2c690e1acb6ada60f140a1651f771a04cc8809bee816040c3.
• Add the complete direct Azure GPT-4o temperature-zero measurement, 1192990eb9c2d1fcf1a71a72fd13c8dc4e5ef4a8c45cbd9c198874a88dc617cc.
• Preserve the three other historical privacy entries. Only pyrit/datasets/scorer_evals/harm/privacy_metrics.jsonl changes.
• Keep scorer/target registrations, preferred names, default objective selection, prompts, rubric, and human labels unchanged. Gold rows changed: 0.

MAE is lower-is-better. Both retained measurements use canonical evaluate_async, the unchanged 93-row, one-human-rater privacy gold set, three trials, dataset version 1.0, and harm-definition version 1.0.

Metric	Registered scorer: before	Registered scorer: refreshed	New direct GPT-4o, temperature 0
MAE	0.373656	0.352151	0.381720
MAE standard error	0.032620	0.031404	0.033053
Model trial-agreement alpha	0.651410	0.799185	0.998094
Determined / input responses	93 / unknown	93 / 93	93 / 93
Trials	3	3	3
Current full-gold constant-median MAE	0.287634, not recorded in legacy entry	0.287634	0.287634

Neither measured configuration beats the constant baseline. The refreshed incumbent's MAE decrease of 0.021505 is within the combined noise margin of approximately 0.0453; the direct configuration's MAE is numerically higher by 0.029570, also within the combined margin of approximately 0.0456. These are exploratory combined-standard-error guards with a 0.02 floor, not significance tests or confidence intervals. The direct configuration was retained by the existing complete/equivalent --write policy without --refresh, but earns no initializer registration. High agreement between model trials is not agreement with human labels.

Identity limits: the registered incumbent retains its exact round-robin identity, with three active Azure leaves; observed responses included GPT-4o snapshots 2024-08-06 and 2024-11-20. The direct temperature-zero configuration used the Japan deployment and snapshot 2024-11-20. It changes routing, deployment/snapshot mix, and temperature, so this is not a controlled temperature-only comparison. The rubric, system prompt, normalization, and aggregation are unchanged. The small, single-rater gold set does not establish human agreement or generalization to other datasets.

Skipped candidate: direct Azure GPT-5.4 failed its no-write, one-trial smoke: only 92/93 rows were determined after one HTTP 400. Its metrics were invalid for complete-coverage comparison; no full evaluation or registry entry was retained. The error subcode was not captured, so no specific failure cause is asserted.

Judge usage: Azure 837 actual transport attempts: 279 smoke attempts across three configurations and 558 full-evaluation attempts across the two retained configurations. There were 836 HTTP 200 responses and one HTTP 400, with no additional retry/fallback attempts. OpenRouter 0. This stays below the 5,000-call run budget and consumes none of the capped OpenRouter budget. Token usage and pricing were not collected; dollar spend and monetary savings are unknown.

Reproduction

Use richlundeen/automations revision bc46e4f2484bd614a6c68804f588e4d78aa1568c, a checkout of this PR, and its synced PyRIT environment with authorized Azure/vault access. Run from the automation root; credentials are loaded in memory. Different deployment routing or model snapshots can produce different measurements.

Save this factory outside the PyRIT checkout as privacy_factory.py:

from pyrit.prompt_target import PromptTarget
from pyrit.score import LikertScalePaths, SelfAskLikertScorer


def build(target: PromptTarget) -> SelfAskLikertScorer:
    return SelfAskLikertScorer.from_likert_scale(
        chat_target=target,
        likert_scale=LikertScalePaths.PRIVACY_SCALE.load(),
    )

$checkout = 'C:\path\to\PyRIT'
$factory = 'C:\path\to\privacy_factory.py'
$python = Join-Path $checkout '.venv\Scripts\python.exe'
$results = Join-Path $env:TEMP 'pyrit-privacy-metrics'
New-Item -ItemType Directory -Path $results -Force | Out-Null
$env:PYTHONPATH = (Get-Location).Path
$env:AZURE_TOKEN_CREDENTIALS = 'AzureCliCredential'
$incumbent = '11412a611f8c19b2c690e1acb6ada60f140a1651f771a04cc8809bee816040c3'

# Smoke, verify complete coverage and identity, then refresh the exact incumbent.
& $python -m automations.update_scorer_metrics.run evaluate --checkout $checkout --scorer likert_privacy_gpt4o --provider azure --incumbent $incumbent --files harm/privacy.csv --harm-category privacy --trials 1 --max-concurrency 5 --output (Join-Path $results 'incumbent-smoke.json')
& $python -m automations.update_scorer_metrics.run evaluate --checkout $checkout --scorer likert_privacy_gpt4o --provider azure --incumbent $incumbent --trials 3 --max-concurrency 5 --write --refresh --output (Join-Path $results 'incumbent-full.json')

# GPT-5.4 smoke was incomplete in this run; no full run was performed.
& $python -m automations.update_scorer_metrics.run evaluate --checkout $checkout --factory $factory --judge-model gpt-5.4 --provider azure --incumbent $incumbent --files harm/privacy.csv --harm-category privacy --trials 1 --max-concurrency 5 --output (Join-Path $results 'gpt54-smoke.json')

# Smoke temperature zero; run full only with complete coverage and the intended identity.
& $python -m automations.update_scorer_metrics.run evaluate --checkout $checkout --factory $factory --judge-model gpt-4o --provider azure --temperature 0 --incumbent $incumbent --files harm/privacy.csv --harm-category privacy --trials 1 --max-concurrency 5 --output (Join-Path $results 'temp0-smoke.json')
& $python -m automations.update_scorer_metrics.run evaluate --checkout $checkout --factory $factory --judge-model gpt-4o --provider azure --temperature 0 --incumbent $incumbent --trials 3 --max-concurrency 5 --write --output (Join-Path $results 'temp0-full.json')

The original run refreshed the incumbent before the variant comparisons; use that fresh entry for comparisons rather than the historical value. GPT-5.4's full run was deliberately skipped. The commands above reproduce the canonical evaluations, not the separate body-free transport-accounting instrumentation.

No deployment is included. The before-survey deployment was at 566c3e24d66b0774c4279997b1dc2b4f1bebf254 and did not register a privacy scorer; no live-metrics change or readback is claimed.

[Copilot]

🟢 Approval recommended

The privacy scorer evaluation-record update has no active review issues.

No open issues 🎉

What changed in this PR

Refreshes PyRIT’s privacy scorer evaluation records with explicit coverage and baseline metrics, and adds a direct Azure GPT-4o temperature-zero comparison. Neither configuration beats the constant-median baseline; no scorer promotion or registration change is included.

Changes:

• Refreshes the registered scorer’s existing evaluation with 93/93 input coverage and updated metrics.
• Adds a complete three-trial temperature-zero evaluation.
• Preserves three historical entries, with no changes to prompts, rubric, or human labels.

File	Description
pyrit/​datasets/​scorer_evals/​harm/​privacy_metrics.jsonl	Refreshes the incumbent measurement and adds the Azure comparison with coverage and baseline metrics.

_{Review effort: Balanced}

[jbolor21]

Richard Lundeen (@richlundeen) none of this is blocking but some questions & future work related to this PR:

Big question on automation here:
Full-dataset "smoke" pass (93 rows × 1 trial): Running 1 trial across all 93 rows, discarding it, and then re-running the same 93 rows for 3 trials effectively runs 4 trials per dataset while screening candidates on the full evaluation set. Not clear on the intent of this smoke pass, as it seems to mix two different goals if I'm understanding correctly:

• Goal A: "Does this endpoint/config work without errors?"
  if so we can tightly decrease number of calls

• If checking basic endpoint/schema health: We can just run 1 trial on a tiny fixed sample (e.g., 5–10 rows).

• If checking that ALL 93 rows complete (93/93, with no row-specific HTTP 400s):* Reuse Trial 1 / fail-fast - run the 3-trial eval directly, abort after Trial 1 if any row errors, and if all 93 succeed, keep those scores as Trial 1 instead of discarding them and running 3 more passes.

• Goal B: "Is this candidate worth a full 3-trial evaluation?" (Screening / tuning)
  here we can also decrease number of calls

• Split the 93 rows once (grouped by objective) into a ~25-row dev set and a ~68-row held-out test set. Screen/compare candidates on the dev set, and only run the final 3-trial evaluation on the held-out test set.

Other PR's work/comments:

• Dev vs. holdout split before tuning: Since likert_privacy_gpt4o is worse than the constant-median guess (0.352 vs 0.288), if we start iterating on prompts/rubrics, screening on all 93 rows will overfit the benchmark. We should consider splitting privacy.csv (grouped by objective) into a small dev/tuning split and a held-out test split. (and we should ensure to do this for all our benchmark sets if we are tuning) (also not blocking on this PR but a future one)

[jbolor21]

Left my comments at the end but none of it is blocking!