Skip to content

BUG: Jinja filters on a template parameter break when a YAML seed loads #4272

Open

Description

@nniiovoo

Describe the bug

When a YAML seed loads, SeedPrompt and SeedObjective render the template once with only the path variables. A filter on a parameter that isn't supplied yet runs on the placeholder. {{ prompt | upper }} turns into {{ PROMPT }}, so the later render with prompt=... fails with 'PROMPT' is undefined. {{ prompt | length }} turns into 0, and filters like replace or trim are just dropped.

default and the is tests already keep the template as it is (#3098), and so do filters that loop over the value, like join. Other filters don't.

Steps/Code to Reproduce

import tempfile
from pathlib import Path

from pyrit.models import SeedPrompt

for value in ["Say {{ prompt | upper }}", "{{ prompt | length }} characters"]:
    with tempfile.TemporaryDirectory() as folder:
        path = Path(folder) / "template.yaml"
        path.write_text(f"value: |-\n  {value}\nparameters:\n  - prompt\n")
        seed = SeedPrompt.from_yaml_file(path)

    print(f"loaded as {seed.value!r}")
    try:
        print("  rendered:", repr(seed.render_template_value(prompt="hello")))
    except ValueError as error:
        print("  render failed:", str(error).split(". Template")[0])

Expected Results

loaded as 'Say {{ prompt | upper }}'
  rendered: 'Say HELLO'
loaded as '{{ prompt | length }} characters'
  rendered: '5 characters'

Actual Results

loaded as 'Say {{ PROMPT }}'
  render failed: Error rendering template '<unnamed template>': 'PROMPT' is undefined
loaded as '0 characters'
  rendered: '0 characters'

None of the bundled templates put a filter on a parameter, so only custom ones run into this. A fix is to wrap every filter at load with the same deferral guard as default (_render_trusted_template_value in seed.py). I have it ready with a test and will open a PR.

Versions

  • OS: macOS 26.4.1 (arm64)
  • Python 3.13.16
  • jinja2 3.1.6
  • PyRIT main (fbef947)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions