Repository navigation
FEAT: Save targets from the Registry (part 9) - #4027
Open
varunj-msft wants to merge 290 commits into
Open
varunj-msft wants to merge 290 commits into
varunj-msft wants to merge 290 commits into
Conversation
…reports an unknown attack type
…s that can't be checked
The banner tied a retry to its failure only through the failed state, so a retry from an earlier failure could still show its answer when the polls missed the runtime becoming ready in between. The generation changes only when the runtime becomes ready or the backend restarts, so a new one now starts a new failure too.
A long failure list made the banner taller than a phone screen, and the app doesn't scroll, so Retry could end up out of reach. The list now scrolls on its own within a third of the screen, and can be scrolled from the keyboard. When the list can't be loaded, the banner says so, keeps the last list it had, and keeps trying.
Scripts a scan runs at launch register their targets, converters and scorers with replace, so one that used a saved entry's name silently replaced it for everyone, and scripts that skip names already taken (the technique and tools scripts, round-robin grouping, the scorer script's preferred scorers) left the scan with the saved entry instead of their own. Restore now marks the entries it rebuilds, the registries and those scripts refuse to replace or stand in for a marked entry, and the launch stops with a 409 naming the script and the entry, even when the script caught the refusal itself. A suite shows the same message for that entry.
A launch script that caught the refusal and raised its own error skipped the recorded-refusal check, so the launch failed with that error instead of the 409 naming the saved entry. The clash is now reported on any error once a refusal was recorded; other errors pass through as before.
# Conflicts: # pyrit/prompt_target/common/prompt_target.py # pyrit/prompt_target/hugging_face/hugging_face_chat_target.py
# Conflicts: # tests/unit/backend/test_runtime_lifecycle.py
# Conflicts: # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_target_service.py
Richard Lundeen (richlundeen)
left a comment
Contributor
There was a problem hiding this comment.
Found six issues to address before approval. Details are on the affected lines.
Saving now refuses what restore can't rebuild: a setting left to the server's environment (each target type declares these, replacing the backend's class-name table) and parameters the create API can't set. A cancelled save finishes or is undone before the cancellation goes on, rollback deletes only the version this save wrote, storage failures log a redacted cause chain, and capability detection keeps the target's origin.
The dialog keeps the type's own reasons from the type list and gets the rest from POST /api/targets/saveability, which runs create's checks and the library's save checks without building or saving anything, so the warning and the save share one policy. The Entra ID endpoint message shows on the Endpoint URL field after it loses focus.
Saved-entry storage failures now pass their traceback through redact_urls, and code scanning flags the scheme pattern as slow on a long run of letters. A scheme is short, so bounding it keeps the match linear.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. It builds on #4026, which saves targets created through the API. Until the earlier PRs merge, the diff here also shows their changes.
Problem. The New Target dialog in Registry > Targets still makes targets that are gone after the backend restarts. It sends no name, so the backend makes one up (
compat_<hex>), and the Targets table doesn't say which targets will come back.Change.
openai_chat_gpt_4o). The model tells saved targets apart and keeps the suggestion off names the startup scripts register, such asopenai_chat. The field checks the same rule the backend uses for saved names. A name the Targets list already has, or one the backend says is taken (409), shows on the field and holds Save until the name changes.persist: true). When a target can't be saved, the button reads Create as Temporary and a warning says why, and that a Temporary target is gone after the backend restarts or is reinitialized. The dialog says so before submitting. The backend's type list tells it when the type can't be saved or the chosen sign-in can't be saved (an API key, for now). For the values entered, such as a typed credential or a round robin with a Temporary member, it asks the newPOST /api/targets/saveability, which runs create's checks and the library's save checks without building or saving anything, so the warning and the save share one policy. The check waits until typing pauses, and only its answer for the current values counts. If Save is pressed first, the backend's refusal turns the button into Create as Temporary for those exact values, and changing any of them goes back to Save Target. No other error offers Temporary.GET /api/targets/typesaddssaveable_auth_modesandnot_saveable_reasonto each type. They come from the checks a save runs (the class check, and the server-setting check for each auth mode with those settings turned off), so the dialog keeps no copy of the credential rules."type": "/errors/not-saveable"), so the dialog doesn't read messages. When the only problem is an endpoint that isn't a full http(s) or ws(s) address, the request gets a plain 400 instead, since a Temporary target with that address wouldn't work either.role="status"so screen readers read it out, and the Entra ID error is the field's validation message. Fluent's message bars only announce through an announcer the app doesn't render.What stays the same. API callers that don't send
persistget Temporary targets as before, and may still leave out the name. Permission checks still come first, and restore is unchanged.Not covered.
Tests and Documentation
tests/unit/registry/test_registry_recipe_rules.py:check_saveable_classlists the auth modes a saved target of each type can use, refuses a class that can never be saved, and agrees with the check a save runs for every registered target type and auth mode.tests/unit/backend/test_target_service.py: the type list says which auth modes a saved target can use and why a type can't be saved; saving a type that can't be saved is refused; an endpoint that isn't a full web address is an error to fix, not a refusal.tests/unit/backend/test_api_routes.py: a refused save is a 400 with"type": "/errors/not-saveable"; other bad requests stay untyped;POST /api/targets/saveabilityanswers with the save checks' reason, and a request creating would refuse gets the same 400 or 403.tests/unit/backend/test_target_service.py(TestCheckSaveability): the check answers as saving would for a Temporary round-robin member, a typed key and Entra ID, without building anything, and refuses a request creating would refuse.CreateTargetDialog.test.tsx(new "CreateTargetDialog saving" block): saving by default with Entra ID or No key, also when the type details arrive late; the suggested name from the type and model until it's edited; names already listed or taken (409) held on the field; when Create as Temporary is offered and why (an API key or a type that can't be saved from the type list; a typed credential or a round robin with a Temporary member from the backend's check, and not from the list's labels), and only after a typed refusal for those exact values; an answer for values that have since changed is ignored; no Temporary offer on 409, 422, 403 or other errors; the warning as a live region; starting over after the dialog closes; the Entra ID error as the Endpoint URL field's description once the field is left.TargetTable.test.tsx: the Saved, From configuration and Temporary labels.registryName.test.tsanderrors.test.ts: the name suggestion and the typed refusal check.frontend/e2e/config.spec.ts(mock): saving through the dialog sends Entra ID andpersist: true.frontend/e2e/registry.spec.ts(seeded, real backend): a target saved from the dialog is still Saved after a reload.python -m pytest -n 8 tests/unit/converter tests/unit/prompt_target tests/unit/registry tests/unit/backend: 11,925 passed, 9 skipped, 4 xfailedcd frontend && npx jest CreateTargetDialog.test.tsx api.test.ts: 227 passedcd frontend && npx playwright test --project=mock e2e/config.spec.ts: 18 passednpx tsc --noEmit,npx eslinton the changed files, and pre-commit (ruff, ty, docs checks): passedPOST /api/targets/saveabilitysays an Entra ID target and a round robin over a saved target can be saved, gives the credential reason for a typed key without echoing it, and refuses a missing model with the same 400 as create.persistcame out Temporary, with the reason shown. After a backend restart the saved targets came back as Saved, the Temporary ones were gone, and a message sent through the restored Entra ID target got an answer. No key was in any saved file.doc/gui/0_gui.md(Registry): saving a target, where saved targets are kept, how to remove one for now, and what happens when a startup script registers the same name.pyrit/backend/README.md: the new type fields, the typed 400, and the saveability check.No JupyText changes.