Hawestra/scenario blog by hannahwestra25 · Pull Request #3 · hannahwestra25/PyRIT

hannahwestra25 · 2026-06-04T16:27:59Z

Description

Tests and Documentation

Covers AttackTechnique abstraction (microsoft#1592), AttackTechniqueRegistry (microsoft#1611), standardized attack args (microsoft#1608), and v0.14.0 scenario improvements including the new TextAdaptive scenario and EpsilonGreedyTechniqueSelector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restructure around the framework/scanner/GUI three-layer story, add a tour of the scenario catalog with a RapidResponse highlight, tighten the v0.13/v0.14 updates section, and shorten the adaptive scenarios section to the demo-script framing (registry + selector + ASR + cross-run learning). Note GUI roadmap for scenarios in the wrap-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove the 'Where scenarios fit' section; fold the scanner mention into the intro. - Drop the colored dots from scenario flavor headers. - Reframe the RapidResponse opener as a broad starter scan / jumping-off point for deeper testing, rather than an urgent-incident scenario. - Expand the v0.13/v0.14 section with concrete usage detail for AttackTechnique, AttackTechniqueRegistry, Better Scenario Tracking, and SequentialAttack. - Audit baseline language so baseline reads as a generic Scenario feature, not a RapidResponse- or adaptive-specific behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T16:30:15Z

+## What's in a scenario
+
+Cracking one open, every scenario bundles four things:
+


what about a strategy ?

Good call — added a new 'Strategies — the runtime knob' bullet here so the strategy/--strategies concept gets introduced up front instead of only appearing in the RapidResponse section.

hannahwestra25 · 2026-06-04T16:32:25Z

+
+### A closer look: `RapidResponse`
+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on where a target is strong and where it's soft, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into the categories or techniques that came back interesting.


"where it's soft" sounds weird

Reworded to 'which categories the target handles well and which ones come back concerning.'

hannahwestra25 · 2026-06-04T16:32:49Z

+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on where a target is strong and where it's soft, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into the categories or techniques that came back interesting.
+
+It crosses two axes: **techniques × harm categories**. The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The harm side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.


don't all scenarios cross two axes ?

True — every scenario crosses techniques × datasets, so that framing was misleading. Dropped 'It crosses two axes' and reworded so the actually-RR-specific bit (results grouped by harm category rather than by technique) is what's called out as the distinguishing feature.

hannahwestra25 · 2026-06-04T16:33:55Z

+
+## What's improved in v0.13.0 and v0.14.0
+
+A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.


Suggested change

A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.

A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.

Applied. Also added a follow-on sentence noting that v0.14 did still ship two new scenarios (RapidResponse and AdversarialBenchmark) so the section doesn't undersell the catalog growth — but the framing is now that machinery was the bulk of the work.

hannahwestra25 · 2026-06-04T16:34:51Z

+
+A lot of the recent work has been less about new scenarios and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.
+
+**A real abstraction for "an attack you can drop into a scenario."** Before v0.13.0 a scenario glued attacks together by hand, knowing how to construct each one and what arguments it needed. [`AttackTechnique`](https://github.com/microsoft/PyRIT/pull/1592) replaced that with a single bundle: the attack strategy class, the `seed_technique` configuration that tells the attack how to mutate prompts (jailbreak template, encoding, role-play wrapper, etc.), plus any technique-specific defaults like which adversarial chat to use. A scenario now composes a *list* of `AttackTechnique`s and hands them to the executor — the scenario doesn't need to know the internals of TAP versus Crescendo versus a converter-based attack, just that it has techniques to run.


remove all PR links

Done — stripped every github.com/.../pull/ link throughout the post.

hannahwestra25 · 2026-06-04T16:36:45Z

+
+**A catalog those techniques live in.** [`AttackTechniqueRegistry`](https://github.com/microsoft/PyRIT/pull/1611) is where techniques register themselves with metadata: name, description, tags like `default` / `single_turn` / `multi_turn` / `light`, modality, what kinds of targets they work against. Scenarios pull techniques out via tag queries — `TagQuery.any_of("default")`, `TagQuery.all_of("multi_turn", "text")` — instead of importing each one by name. The scanner CLI uses the same registry to list and describe what's available (`pyrit_scan list-techniques`). And to add your own, you write a factory and register it with a tag; every scenario that queries by that tag picks it up for free. Attack arguments were [standardized in the same release](https://github.com/microsoft/PyRIT/pull/1608), which is what lets the registry instantiate techniques generically — every constructor speaks the same dialect.
+
+**Attribution that survives runs.** [Better Scenario Tracking](https://github.com/microsoft/PyRIT/pull/1758) added a scenario-run ID that gets stamped onto every `AttackResult` row the run produces. That sounds small but unlocks a lot: when you resume a partially-completed scenario (via `scenario_result_id`), the framework can ask memory "which objectives already have results for this run?" and skip them without double-counting; cross-run analytics like "how did `RedTeamAgent` do on this target across our last ten scans?" stop needing manual labeling; and the printer can roll results up to the correct scenario invocation instead of mixing in unrelated history sitting in the same database. This is also what makes the adaptive selector's cross-run learning trustworthy — it can scope its history queries cleanly.


it might be helpful to have a diagram of how these interact

Added a mermaid diagram inline in the Better Scenario Tracking paragraph showing scenario_run_id getting stamped onto AttackResult rows and flowing through memory to resume / cross-run analytics / printer rollup / adaptive selector scoping.

hannahwestra25 · 2026-06-04T16:36:58Z

+
+**Sorted breakdowns and a new compound primitive.** Scenario printers now [sort the per-group breakdown by success rate](https://github.com/microsoft/PyRIT/pull/1809), so the categories the target is most vulnerable on float to the top of the report instead of being buried alphabetically. And [`SequentialAttack`](https://github.com/microsoft/PyRIT/pull/1819) shipped as a compound `AttackStrategy`: give it a list of attacks and a `SequenceCompletionPolicy` (`FIRST_SUCCESS` stops as soon as one lands, `EXHAUSTIVE` runs them all, `FIRST_DECISIVE` stops on the first clear pass-or-fail), and it runs them in priority order. Adaptive scenarios use this under the hood, but it's available as a standalone building block any time you want "try the cheap thing first, escalate only if needed."
+
+**Smaller polish worth knowing about.** `include_baseline` moved from the `Scenario` constructor to `initialize_async` ([#1700](https://github.com/microsoft/PyRIT/pull/1700)) so you decide at run time, not construction time. `BASELINE_POLICY` was renamed to `BASELINE_ATTACK_POLICY` ([#1763](https://github.com/microsoft/PyRIT/pull/1763)) so the name says what it controls. The `FoundryScenario` alias was removed in favor of `RedTeamAgent` ([#1623](https://github.com/microsoft/PyRIT/pull/1623)). And — possibly the most useful change for new users — [scenario doc pages for the seven scenarios that didn't have them](https://github.com/microsoft/PyRIT/pull/1558) finally exist.


hannahwestra25 · 2026-06-04T16:37:39Z

+
+Three pieces make it work:
+
+- **The registry** — the same `AttackTechniqueRegistry` from v0.13.0. It's the catalog of available techniques the scenario can pick from.


mention something about asr ?

Pulled ASR into the opening paragraph of the adaptive section so it's introduced before the 'three pieces make it work' bullets (which now reference ASR by its acronym instead of spelling it out for the first time).

hannahwestra25 · 2026-06-04T16:38:30Z

+
+- The scenarios docs landing page: [`doc/code/scenarios/`](../code/scenarios/0_scenarios.ipynb).
+- The end-to-end walkthroughs from the scanner side: [`pyrit_scan`](../scanner/1_pyrit_scan.ipynb) and [`pyrit_shell`](../scanner/2_pyrit_shell.md).
+- The adaptive scenarios notebook (shipped alongside `TextAdaptive`) is the fastest way to see the bandit in action against a real target.


add link to notebook

Done — pointed the bullet at doc/code/scenarios/3_adaptive_scenarios.ipynb.

- Drop 'two people running the same pass...' sentence; switch opener to 'Enter scenarios.' - Add a 'Strategies - the runtime knob' bullet so the runtime-selection concept is introduced before it's used in RapidResponse / adaptive sections. - Add adaptive scenarios as a fifth flavor in the catalog with a link to its section; mark RapidResponse and AdversarialBenchmark as new in v0.14. - Rework RapidResponse closer-look: replace 'where it's soft' phrasing, drop the 'crosses two axes' framing (every scenario does that), keep the grouping detail as the actually-RR-specific bit. - Apply the 'less about building out our scenario library' wording, then explicitly surface the two new v0.14 scenarios so the section isn't misleading about scope. - Strip every github.com PR link. - Add two new paragraphs: 'Configuration from the CLI and from YAML' and 'Parallel execution within a scenario' to cover microsoft#1680 and microsoft#1783. - Add a mermaid diagram to the Better Scenario Tracking paragraph showing scenario_run_id flowing into memory and out to resume / analytics / printer / adaptive selector. - Drop the 'Smaller polish worth knowing about' paragraph. - Pull ASR mention into the adaptive section's opening paragraph. - Point the 'Where to go next' bullet at the real adaptive notebook path (doc/code/scenarios/3_adaptive_scenarios.ipynb). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…, and TextAdaptive bandit trail Three TODO IMAGE blocks added (all HTML-commented so the docs build stays green): 1. After the 'There are five flavors' opener: pyrit_scan list-scenarios catalog screenshot (2026_06_04_scan_list_scenarios.png). 2. End of the RapidResponse 'A closer look' section: ConsoleScenarioResultPrinter summary showing baseline + per-harm-category breakdown sorted by success rate (2026_06_04_rapid_response_output.png). 3. After the TextAdaptive code block: per-objective technique trail + wins/picks/ rate summary, captured from doc/code/scenarios/3_adaptive_scenarios.ipynb (2026_06_04_text_adaptive_output.png). To activate any of them: drop the PNG into doc/blog/ with the indicated filename and uncomment the markdown image line inside the comment block. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ructure Moved the diagram from the 'Attribution that survives runs' section (where the surrounding prose already covers the same ground) to the 'A catalog those techniques live in' paragraph, where it explains the v0.13 abstraction story visually: techniques register with tags into AttackTechniqueRegistry, TagQuery subsets become ScenarioStrategy enum members, --strategies picks one at run time, and the scenario fans the chosen techniques out across its datasets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T18:57:09Z

+
+<small>4 Jun 2026 - Hannah Westra</small>
+
+When we first introduced scenarios in PyRIT a few releases back, the pitch was pretty simple: most of the operators we talked to were assembling the same set of pieces — a target, a curated dataset of objectives, a few attack strategies, a scorer — and running them in a loop. The framework gave them all the right Lego bricks, but every team was clicking those bricks together by hand. There was no clean unit of work to point at when you wanted to say, "run this exact configuration, then compare it against that one."


is it obvious who operators are ? is that a common term for red teamers ? do we use that terminology in the rest of the repo / website ?

Good catch. operator is only used once in the broader docs and never with much consistency — the repo otherwise leans on red teamer / red teaming. Swapped both occurrences (the opening paragraph and the strategies bullet) so the terminology is consistent with framework.md and the rest of the site.

hannahwestra25 · 2026-06-04T19:05:21Z

+
+Cracking one open, every scenario bundles five things:
+
+- **Techniques — the *how*.** How are we going to attack? Maybe we just send the prompt directly. Maybe we wrap it in a role-play scenario. Maybe we escalate over multiple turns with Crescendo or TAP. Techniques include the attack strategy plus its converters, jailbreak templates, and adversarial-chat configuration — basically all the knobs that affect how the attack is crafted and delivered.


i don't think role-play is a scenario ? this is confusing.

You're right — scenario is already a defined term in this post, so saying wrap it in a role-play scenario overloads it mid-sentence. Changed to wrap it in a role-play prompt template which matches the actual mechanism (RolePlayAttack with a RolePlayPaths template) without colliding with the scenario terminology.

hannahwestra25 · 2026-06-04T19:06:16Z

+
+Scenarios can also opt in to running a **baseline pass** — sending the raw, unmodified prompts as a control group so the report can show "what the model does without any attack" next to "what the model does once attacked." Baseline behavior is generic to the `Scenario` base class (toggled per run via `include_baseline` and governed by `BASELINE_ATTACK_POLICY`); individual scenarios just decide whether it makes sense for their workload.
+
+The whole point is that you don't have to wire any of this up yourself. Pick a scenario, point it at a target, and the scenario handles the rest.


add a screenshot of a pyrit_scan command to run a scenario and explain what it's doing

Done. Added a fourth screenshot placeholder right after the pick a scenario, point it at a target paragraph: 2026_06_04_scan_run_scenario.png. The capture instructions in the placeholder ask for the command at the top + the initializer log lines + the first technique progress bars (the final summary is already captured in 2026_06_04_rapid_response_output.png, so this one is specifically the what does it look like to kick one off frame). I also added an explanatory paragraph immediately after the image that walks through what the one CLI command actually does end-to-end (initializers populate registries -> scenario lookup -> target resolution -> run_async).

hannahwestra25 · 2026-06-04T19:16:16Z

+
+`RapidResponse` is the broadest of the AIRT scenarios — a comprehensive sweep across the most common techniques and the full AIRT harm-category catalog. It's a natural jumping-off point: you run it to get a wide, shallow read on which categories the target handles well and which ones come back concerning, then pivot to the more focused AIRT scenarios (or to `TextAdaptive`) to dig into whatever came back interesting.
+
+The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The dataset side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.


i think the "technique side pulls" wording is confusing here

Agreed, the technique side / dataset side framing was clunky. Collapsed it into one sentence: It runs seven core techniques (...) across seven AIRT datasets (...). That gets both lists across without the awkward two-axis personification.

hannahwestra25 · 2026-06-04T19:18:28Z

+
+The technique side pulls from seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`). The dataset side covers seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.
+
+Like other scenarios it can run a baseline pass first (`include_baseline=True`, the default) — the raw prompts with no converters or wrapping techniques — so the report has a control group sitting next to the attacked numbers. Then it runs every selected technique against every selected dataset, in parallel, through the execution engine. The thing that's specific to `RapidResponse` is the *grouping*: results roll up by harm category rather than by technique, because when leadership asks "are we exposed to hate speech?", you want the answer organized that way.


explain what grouping by harm category vs technique means. are other scenarios grouped by technique? I think we can omit the part about baseline here.

Both done. (a) Removed the baseline mention from this paragraph — baseline is a generic Scenario feature and is already covered in the What's in a scenario section a few paragraphs up, so re-explaining it here was redundant. (b) Yes, the default base Scenario._build_display_group returns the technique name, so most scenarios print one block per technique; RapidResponse overrides _build_display_group to return the dataset (seed_group_name) instead. Rewrote the paragraph to spell that out concretely with two example formats (crescendo_simulated: 14/50, role_play: 7/50 vs airt_hate: 21/100, airt_violence: 4/100) and a one-liner noting it's the same AttackResult rows in memory, just a different rollup axis on the way out.

hannahwestra25 · 2026-06-04T19:19:34Z

+
+The full path from a single technique to a scenario run looks like:
+
+```mermaid


diagram sucks. it should lay out how the attack technique relates to attack strategies relates to a scenario relates to an atomic attack etc.

Fair. Rewrote the diagram to cover the full chain instead of just the registry-to-strategy slice. It now shows AttackStrategy + seed configuration -> AttackTechnique -> AttackTechniqueRegistry (with the ScenarioTechniqueInitializer arrow making it explicit who populates the registry) -> ScenarioStrategy -> Scenario (+ its datasets feeding in) -> AtomicAttack -> ScenarioResult. Switched to flowchart TB since there are too many levels for an LR layout to stay readable. Also added a short follow-on paragraph that defines AttackStrategy, AttackTechnique, and AtomicAttack in plain English since the diagram now uses all three terms.

hannahwestra25 · 2026-06-04T19:20:33Z

@@ -0,0 +1,176 @@
+# Scenarios: Where We Started, and Where We're Going
+


should we incorporate initializers somewhere ? also i think i want more on the why of scenarios and rapid response and adaptive scenarios

Two things in this one: (a) Initializers. Added them in two places. They get a sentence in the A catalog those techniques live in paragraph (Population of the registry itself is handled by initializers — the canonical one is ScenarioTechniqueInitializer...), show up explicitly on the registry arrow in the rebuilt mermaid diagram, and the new pyrit_scan paragraph after line 23 walks through how ScenarioTechniqueInitializer, TargetInitializer, and LoadDefaultDatasets populate their registries before the scenario itself runs. (b) More on the why. Expanded the opening paragraph (the Lego bricks paragraph) to call out the concrete pains scenarios solved — rerun after a model update, hand a config to a teammate, diff against last quarter — instead of the vague no clean unit of work. Expanded the RapidResponse opener to name the trigger moments (new model onboarded, new model release, new vulnerability in the news) and what the question being asked actually is at those moments. Expanded the adaptive opener to make the cost of brute force concrete (latency, API rate-limit budget, adversarial-chat tokens) so the motivation isn't just wasted attempts in the abstract.

- Replace 'operators' with 'red teamers' to match repo terminology. - Reword 'role-play scenario' bullet to 'role-play prompt template' so the word scenario isn't overloaded mid-sentence. - Add a fourth screenshot placeholder + explanatory paragraph showing what 'pyrit_scan airt.rapid_response --target my_target' actually does at the CLI; this is also where the post introduces initializers (ScenarioTechnique- Initializer, TargetInitializer, LoadDefaultDatasets) as the things that populate the registries before run_async fires. - Reword the RapidResponse 'technique side pulls / dataset side covers' sentence as a single 'It runs N techniques across N datasets' line. - Drop the baseline mention from the RapidResponse paragraph (already covered in 'What's in a scenario') and replace it with a concrete explanation of what 'group by harm category vs by technique' means in the printer. - Rebuild the mermaid diagram. It now spans the full chain: AttackStrategy + seed config -> AttackTechnique -> AttackTechniqueRegistry (via Scenario- TechniqueInitializer) -> ScenarioStrategy -> Scenario + datasets -> AtomicAttack -> ScenarioResult, with a follow-on paragraph that defines AttackStrategy, AttackTechnique, and AtomicAttack in plain English. - Expand the opening paragraph and adaptive intro slightly with more 'why' (reproducibility, comparing runs, real cost of brute-force scans). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T19:33:07Z

+![pyrit_scan list-scenarios output](2026_06_04_scan_list_scenarios.png)
+-->
+
+**Foundry — `RedTeamAgent`.** The integration with [Azure AI Foundry's red-teaming library](https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/red-team-tools). Organized by complexity (easy / moderate / difficult) rather than harm type. Easy = converters like Base64, ROT13. Difficult = multi-turn attacks like TAP and Crescendo. Built on HarmBench with 25+ techniques. The "throw everything at the wall" approach.


this link doesn't work

Swapped to the working ai-red-teaming-agent doc URL — confirmed reachable. Pushed in 05f4584.

hannahwestra25 · 2026-06-04T19:34:25Z

+![pyrit_scan list-scenarios output](2026_06_04_scan_list_scenarios.png)
+-->
+
+**Foundry — `RedTeamAgent`.** The integration with [Azure AI Foundry's red-teaming library](https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/red-team-tools). Organized by complexity (easy / moderate / difficult) rather than harm type. Easy = converters like Base64, ROT13. Difficult = multi-turn attacks like TAP and Crescendo. Built on HarmBench with 25+ techniques. The "throw everything at the wall" approach.


remove "The "throw everything at the wall" approach."

Dropped the throw everything at the wall sentence.

hannahwestra25 · 2026-06-04T19:34:43Z

+- `ContentHarms` — general-purpose; covers hate, violence, fairness, sexual content
+- `RapidResponse` *(new in v0.14)* — broad starter scan; gets its own paragraph below


remove content harms. it is replaced with rapid response

Removed the ContentHarms bullet from the AIRT catalog. Confirmed in code it's deprecated and aliased to RapidResponse (scheduled removal in v0.15.0), so it shouldn't go on the user-facing list.

hannahwestra25 · 2026-06-04T19:35:10Z

+- `RapidResponse` *(new in v0.14)* — broad starter scan; gets its own paragraph below
+
+**Adaptive — `TextAdaptive`** *(new in v0.14)***.** A new family of scenarios that pick techniques on the fly based on what's worked before. Big enough to get its own section [below](#adaptive-scenarios-v0140).
+


make sure there is no reference to deprecated classes / features

Audited the post — only ContentHarms was deprecated (handled in the bullet above). Everything else in the post is current: seed_technique is still the parameter name on AttackTechnique, include_baseline still exists on initialize_async, BASELINE_ATTACK_POLICY / SequentialAttack / SelectorScope / EpsilonGreedyTechniqueSelector are all v0.13/v0.14 APIs, and FoundryScenario only appears as a one-line historical reference noting it was renamed to RedTeamAgent.

hannahwestra25 · 2026-06-04T19:37:19Z

+
+The whole point is that you don't have to wire any of this up yourself. Pick a scenario, point it at a target, and the scenario handles the rest.
+
+Concretely, the simplest run from the CLI looks like:


Suggested change

Concretely, the simplest run from the CLI looks like:

Concretely, the simplest run to create a RapidResponse scenario from the CLI looks like:

Applied verbatim.

hannahwestra25 · 2026-06-04T19:42:08Z

+flowchart TB
+    AS["AttackStrategy<br/>(the executable algorithm:<br/>PromptSendingAttack,<br/>CrescendoAttack, TAPAttack, ...)"]
+    SC["seed configuration<br/>(jailbreak template, converters,<br/>adversarial-chat target, ...)"]
+    AT["AttackTechnique<br/>(strategy + seed config bundle)"]


some of the words in the diagram are getting cut off

Shortened the node labels (AttackStrategy, seed configuration, etc. now stand alone instead of carrying parenthetical example lists) and shortened the edge labels too (registered (with tags), TagQuery, --strategies, technique × dataset, AttackExecutor). The bullet list right below the diagram now carries all the detail the labels used to.

hannahwestra25 · 2026-06-04T19:42:55Z

+
+## What's improved in v0.13.0 and v0.14.0
+
+A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one. Two new scenarios did join the catalog along the way — `RapidResponse` and `AdversarialBenchmark`, both in v0.14 — but the bulk of the work was under the hood.


Suggested change

A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one. Two new scenarios did join the catalog along the way — `RapidResponse` and `AdversarialBenchmark`, both in v0.14 — but the bulk of the work was under the hood.

A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one.

Applied verbatim — dropped the Two new scenarios did join the catalog sentence.

hannahwestra25 · 2026-06-04T19:44:00Z

+
+A lot of the recent work has been less about building out our scenario library and more about making the underlying machinery sharper, so adding scenarios (and adding the *next* layer of capability on top of them) doesn't require rewriting the last one. Two new scenarios did join the catalog along the way — `RapidResponse` and `AdversarialBenchmark`, both in v0.14 — but the bulk of the work was under the hood.
+
+**A real abstraction for "an attack you can drop into a scenario."** Before v0.13.0 a scenario glued attacks together by hand, knowing how to construct each one and what arguments it needed. `AttackTechnique` replaced that with a single bundle: the attack strategy class, the `seed_technique` configuration that tells the attack how to mutate prompts (jailbreak template, encoding, role-play wrapper, etc.), plus any technique-specific defaults like which adversarial chat to use. A scenario now composes a *list* of `AttackTechnique`s and hands them to the executor — the scenario doesn't need to know the internals of TAP versus Crescendo versus a converter-based attack, just that it has techniques to run. Standardized attack arguments shipped in the same release, which is what lets every technique constructor speak the same dialect.


i don't like how this is laid out with a clause to start the paragraph. same with the paragraph after.

Restructured the whole What's improved section. All six bold lead-in paragraphs are now proper ### subsections (A real abstraction for attacks in a scenario, A catalog those techniques live in, Configuration from the CLI and from YAML, Parallel execution within a scenario, Attribution that survives runs, Sorted breakdowns and a new compound primitive), with the opening sentences rewritten so they read as prose under each heading instead of as clauses dangling off bold text. Defaulted to applying it consistently to all six rather than just the two you flagged — let me know if you'd rather I revert the other four.

hannahwestra25 · 2026-06-04T19:47:04Z

+Three pieces make it work:
+
+- **The registry** — the same `AttackTechniqueRegistry` from v0.13.0. It's the catalog of available techniques the scenario can pick from.
+- **The selector** — the brain. `EpsilonGreedyTechniqueSelector` decides what to try next using an explore/exploit tradeoff: most of the time it picks the technique with the best historical ASR; some of the time it tries something random to make sure it isn't missing a better option. New techniques get a fair shot before the selector settles on favorites.


explain that this is just the default selector but can be configured

Added a note to that bullet: EpsilonGreedyTechniqueSelector is the default, but TechniqueSelector is a small one-method protocol so you can drop in a contextual bandit, Thompson sampler, or any other policy by passing your own implementation to the scenario's selector argument.

hannahwestra25 · 2026-06-04T19:47:37Z

+![TextAdaptive bandit picking techniques per objective and the wins/picks summary](2026_06_04_text_adaptive_output.png)
+-->
+
+`prompt_sending` is intentionally excluded from the adaptive technique pool — since baseline is a generic scenario feature, `TextAdaptive` reuses it as the no-attack control (via `BASELINE_ATTACK_POLICY=Enabled`) so the report still shows the honest "no-attack" number alongside the adaptive results.


Removed the prompt_sending baseline paragraph.

hannahwestra25 · 2026-06-04T19:48:31Z

+    AA -->|"AttackExecutor runs<br/>many in parallel"| Res["ScenarioResult"]
+```
+
+A few things worth pulling out: `AttackStrategy` is the *executable* attack class (e.g. `CrescendoAttack`) — the algorithm itself. `AttackTechnique` wraps a strategy *plus* its seed configuration (templates, converters, adversarial chat) into the unit the registry tracks. `AtomicAttack` is the runnable pairing of one technique with one dataset — what the executor actually executes — and is also the unit at which results get tracked, resumed, and labeled in memory. Each scenario builds its own `ScenarioStrategy` enum from the registry at import time, so the strategy names you see on `--strategies` are always in sync with whatever techniques are currently tagged for that scenario.


put each definition on a new line

Converted the paragraph to a four-bullet list — AttackStrategy, AttackTechnique, ScenarioStrategy, AtomicAttack each get their own line.

- Apply CLI command suggestion (line 25 wording) - Fix broken Azure AI Foundry red-teaming link; drop `throw everything` phrasing - Remove ContentHarms catalog bullet (deprecated in favor of RapidResponse) - Remove RapidResponse strategy-tags paragraph - Restructure `What's improved in v0.13.0 and v0.14.0` section into proper ### subsections instead of bold lead-in clauses, and drop the `Two new scenarios` sentence per suggestion - Shorten mermaid diagram node and edge labels so they stop getting cut off - Convert post-diagram definitions paragraph to a bullet list - Note EpsilonGreedyTechniqueSelector is the default and TechniqueSelector is pluggable - Remove prompt_sending baseline paragraph from the adaptive section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T20:00:05Z

+<!-- TODO IMAGE: Run `pyrit_scan list-scenarios` and screenshot the output, which
+     shows the full catalog (Foundry, Garak, Benchmark, AIRT, Adaptive) with
+     descriptions. Save to doc/blog/2026_06_04_scan_list_scenarios.png and
+     uncomment the line below.
+![pyrit_scan list-scenarios output](2026_06_04_scan_list_scenarios.png)
+-->


Removed the entire list-scenarios TODO IMAGE block.

hannahwestra25 · 2026-06-04T20:00:24Z

+
+It runs seven core techniques (`prompt_sending`, `role_play`, `many_shot`, `TAP`, `crescendo_simulated`, `red_teaming`, `context_compliance`) across seven AIRT datasets (`airt_hate`, `airt_fairness`, `airt_violence`, `airt_sexual`, `airt_harassment`, `airt_misinformation`, `airt_leakage`). By default it sends four prompts per dataset, configurable with `--max-dataset-size`.
+
+The thing that's distinctive about `RapidResponse` is how it groups results in the printer. The base `Scenario` class defaults to grouping by technique — most scenarios print one block per technique ("crescendo_simulated: 14/50 succeeded, role_play: 7/50, …"), which is the right framing when you're asking "which attack is hitting?" `RapidResponse` overrides that to group by harm category instead — one block per dataset ("airt_hate: 21/100, airt_violence: 4/100, …"), summed across all techniques — because when leadership asks "are we exposed to hate speech?", that's the rollup they want. Same `AttackResult` rows in memory, different axis on the way out.


Removed the thing that's distinctive about RapidResponse paragraph. The RapidResponse section is now just the what it covers framing + the seven techniques × seven datasets line.

hannahwestra25 · 2026-06-04T20:00:42Z

+<!-- TODO IMAGE: Run RapidResponse against a target end-to-end (e.g.
+     `pyrit_scan airt.rapid_response --target my_model --strategies default`)
+     and screenshot the final ConsoleScenarioResultPrinter summary. The ideal
+     screenshot shows the baseline column next to the attacked columns and the
+     per-harm-category breakdown sorted by success rate — that one image
+     captures the RR output shape, the new sorted breakdown, and the
+     harm-category grouping discussed in this section all at once. Save to
+     doc/blog/2026_06_04_rapid_response_output.png and uncomment the line below.
+![RapidResponse printer output: per-harm-category breakdown sorted by success rate](2026_06_04_rapid_response_output.png)


Removed the entire rapid_response_output TODO IMAGE block.

hannahwestra25 · 2026-06-04T20:04:26Z

+
+### Configuration from the CLI and from YAML
+
+v0.14 added a generic mechanism for setting scenario parameters at run time — both from `pyrit_scan` arguments (`--max-dataset-size 10 --strategies multi_turn`) and from YAML config files passed with `--config`. The same `set_params_from_args` plumbing is exposed in Python too, so a notebook user can stash their parameters in YAML and load the same config the CLI does. This is what made parameter-heavy scenarios like `TextAdaptive` (selector, epsilon, max attempts per objective, scope) feasible to drive from the CLI.


make sure all version match 0.x.0 so something like 0.13.0 or 0.14.0 with no v before it

Normalized every version reference to 0.x.0 (no v prefix). Updates: v0.10.0 / v0.11.0 -> 0.10.0 / 0.11.0, v0.13.0 -> 0.13.0 (all occurrences), v0.14 and v0.14.0 -> 0.14.0 (all occurrences). Also updated the cross-reference anchor from #adaptive-scenarios-v0140 to #adaptive-scenarios-0140 to match the new heading.

hannahwestra25 · 2026-06-04T20:05:53Z

+### Sorted breakdowns and a new compound primitive
+
+Scenario printers now sort the per-group breakdown by success rate, so the categories the target is most vulnerable on float to the top of the report instead of being buried alphabetically. And `SequentialAttack` shipped as a compound `AttackStrategy`: give it a list of attacks and a `SequenceCompletionPolicy` (`FIRST_SUCCESS` stops as soon as one lands, `EXHAUSTIVE` runs them all, `FIRST_DECISIVE` stops on the first clear pass-or-fail), and it runs them in priority order. Adaptive scenarios use this under the hood, but it's available as a standalone building block any time you want "try the cheap thing first, escalate only if needed."


remove this section

Removed the Sorted breakdowns and a new compound primitive subsection entirely.

hannahwestra25 · 2026-06-04T20:06:44Z

+
+## Adaptive scenarios (v0.14.0)
+
+`RapidResponse` is thorough — but it's brute force. It runs every technique against every objective, and most of those attempts are wasted: maybe Crescendo works great on your target and `prompt_sending` never gets through. You're paying for the misses anyway in latency, API rate-limit budget, and adversarial-chat tokens — and on a wide scan against a real target those costs are not theoretical.


Suggested change

`RapidResponse` is thorough — but it's brute force. It runs every technique against every objective, and most of those attempts are wasted: maybe Crescendo works great on your target and `prompt_sending` never gets through. You're paying for the misses anyway in latency, API rate-limit budget, and adversarial-chat tokens — and on a wide scan against a real target those costs are not theoretical.

`RapidResponse` is thorough — but it's brute force. It runs every technique against every objective, and which can result in wasted attempts: maybe Crescendo works great on your target and `prompt_sending` never gets through. You're paying for the misses anyway in latency, API rate-limit budget, and adversarial-chat tokens — and on a wide scan against a real target those costs are not theoretical.

Applied verbatim.

hannahwestra25 · 2026-06-04T20:08:22Z

+![TextAdaptive bandit picking techniques per objective and the wins/picks summary](2026_06_04_text_adaptive_output.png)
+-->
+
+Bottom line: **`RapidResponse` tells you "here's how every technique did" against this target. `TextAdaptive` tells you "here's the fastest path to breaking this model."** Both are useful; you reach for them at different moments.


i want this to wrap up more cleanly

Reworked the closing. Dropped the bottom line framing and replaced it with a short practical handoff: run RapidResponse first to map exposure, then reach for TextAdaptive on the categories that came back interesting, and every scan sharpens the next because they share the registry + ASR history. Reads more like a wrap-up and flows straight into Where to go next.

- Add the captured TextAdaptive screenshot (per-objective trail + wins/picks summary) and remove the corresponding TODO IMAGE block - Remove the list-scenarios TODO IMAGE block - Remove the RapidResponse 'distinctive grouping' paragraph and the RapidResponse TODO IMAGE block - Remove the 'Sorted breakdowns and a new compound primitive' subsection - Apply the brute-force opener suggestion verbatim on the adaptive section intro - Rewrite the closing wrap-up so the post ends on a tighter practical handoff - Normalize all version references to 0.x.0 (strip 'v' prefix everywhere) and update the cross-reference anchor accordingly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannahwestra25 · 2026-06-04T20:17:08Z

+
+<!-- TODO IMAGE: Capture a terminal session running `pyrit_scan airt.rapid_response
+     --target my_target` (or a similar short scenario). The most useful frame
+     shows the command at the top, the initializer log lines loading
+     techniques / targets / datasets, and the first technique progress bars
+     starting up. The final summary is already captured in
+     2026_06_04_rapid_response_output.png — this image is about the
+     "what does it look like to kick one off" moment. Save to
+     doc/blog/2026_06_04_scan_run_scenario.png and uncomment the line below.
+![pyrit_scan launching a scenario from the CLI](2026_06_04_scan_run_scenario.png)
+-->


Removed the entire scan_run_scenario TODO IMAGE block. That was the last placeholder remaining — the post no longer has any unfilled image slots.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fixes the validate-docs pre-commit failure that flagged blog/2026_06_04_scenarios.md as an orphaned file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ra/scenario_blog

Update the blog post to reflect the actual 10 core-tagged techniques registered in ScenarioTechniqueInitializer: adds pair, and the three persona-driven crescendo variants (movie_director, history_lecture, journalist_interview). Removes prompt_sending which is baseline, not a registered technique. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Replace screenshot with text code block (comment 1) - Fix wording: 'kick off a scan' instead of 'create a scenario' (comment 2) - Fix CLI flag --config to --config-file; clarify which TextAdaptive params are CLI-reachable vs Python-only (comment 3) - Lowercase 'Better Scenario Tracking' to avoid proper-noun read (comment 4) - Add qualifier that not all scenarios expose the same strategy set; move light to parenthetical example (comment 5) - Fix AIRT scenario count from seven to six (comment 6) - Clarify default strategy runs role_play + many_shot + baseline; full core pool available via --strategies single_turn/multi_turn (comment 7) - Remove unused text_adaptive_output.png image Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Comment 7: Don't hard-code technique count (it'll drift). Removed 'currently ten' and the full enumerated list; now describes the pool as 'every core-tagged factory in the registry' and lists specific techniques only in the parenthetical for the full-pool strategies. Comment 3: Drop 'parameter-heavy scenarios like TextAdaptive' framing (it only has one CLI param). Rewrite to clearly say which knobs go through supported_parameters()/CLI vs Python constructor. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ra/scenario_blog

…date Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

hannahwestra25 and others added 3 commits June 4, 2026 11:28