Repository navigation
[Proposal] Optional test generation step bridging SDD and TDD #900
Description
Hi, my team and I have adopted OpenSpec as our standard development workflow — it's been instrumental in helping us
ship reliable code faster.
As we've scaled our usage, one area where we still spend significant time is code review and verification. The specs
already capture what the system should do in a structured, reviewable way, which feels like a natural foundation for
generating tests too — if we can extract requirements from specs and expand them into test scenarios before coding
begins, verification becomes largely automated and traceable back to the spec.
The idea is that this would be an entirely optional step in the workflow:
propose → [test-plan → review → test-write → review] → apply → archive
Developers who don't need it skip it — zero behavior change. For those who want it, the specs they've already written
become the source of truth for both implementation and verification, reducing the manual review burden.
Rough concept
- A planning step that reads specs, extracts requirements (RFC 2119 keywords, Given/When/Then), expands scenarios
across multiple test dimensions, and produces a reviewable plan with traceable scenario IDs - A writing step that turns the approved plan into executable test files in the project's test directory, tagged with
those IDs for traceability - Optionally, apply could track test progress during implementation when a test plan exists
The scenario IDs would trace across plan → test code → traceability report, so reviewers can verify coverage at a
glance.
Questions
I saw the discussion in #557 about keeping OpenSpec lightweight by default while making it configurable for power
users through the schema system. Would something like this be better suited as:
- Built-in optional workflows (in
ALL_WORKFLOWS, notCORE_WORKFLOWS)? - A custom schema extension?
- Something else?
I'm currently working on adding these optional steps for our own use. I'd love to know if this is a direction you're
interested in expanding toward — happy to share my current draft as a starting point for discussion.
And if this isn't something you'd consider adding, I'd genuinely appreciate hearing your concerns or the
decision behind not going in this direction — it would help me spot blind spots in my own thinking. Also curious
about where you see the customization surface heading — are there plans to extend schema customization further (e.g.,
beyond artifacts and instructions), or is the current scope roughly where it'll stay?
Thanks!
Activity
[-][Proposal] Optional test generation step — bridging SDD and TDD[/-][+][Proposal] Optional test generation step bridging SDD and TDD[/+]awaragi commentedon Apr 1, 2026
I think this is a very good proposition. I am a big fan of ATDD and extracting acceptance criteria from spec is relatively straight forward and then it becomes of a design/planning stage to align some acceptance tests in e2e vs unit tests.
alfred-openspec commentedon Apr 5, 2026
Thanks for raising this. Closing the loop between specs, code, and tests is directionally where we want to go, but we want to scope it properly after higher-priority workflow pieces like multi-repo and monorepo support land first.
If you’re open to sharing your draft, that would be genuinely useful input when we get there.
guillaume-flambard commentedon Sep 10, 2026
I built the verification half of this and measured it on real corpora, so here is
some data rather than an opinion. It may be useful whenever this gets scoped.
I wanted to answer one question deterministically: which
#### Scenario:in aspec has a test in front of it? No test generation, no LLM, no running the suite.
Just the specs and the statically-extracted titles of Vitest and Jest tests.
Automatic linking by text similarity does not work, and that matters for the
design above.
I started with a Jaccard score over normalised words, expecting it to carry most
of the linking. Measured on five public repositories that use OpenSpec, 8,860
criteria in total:
None of those repositories could have linked anything by hand: the convention I
used did not exist when they were written. The table measures what a similarity
score recovers unaided, and says nothing about the repositories themselves.
Two findings.
Where specs and tests are written in the same language, similarity finds real
links but few of them: 2 to 10 percent of scenarios. The passing matches are
solid, sharing four to five significant words on median, and none looked wrong on
inspection. Most of the rest land in a middle band, and reading a sample says
roughly half of those are the right test sitting just below the threshold
because the test title carries extra words.
Where the languages differ, similarity is inert. On a French-spec repository with
English test titles, 268 of 482 failures shared not one significant word with
any of 644 test titles. A word-based score does not cross a language boundary,
and no threshold fixes that.
So the explicit, traceable scenario ID in the proposal above is not a nicety,
it is the mechanism. Anything that hopes to infer the link after the fact will
recover a small fraction of it. I would push that further: if the ID lives in the
spec, it survives the test being renamed, which inference never does.
One detail I ran into that is worth knowing if IDs ever get specified here: any
metadata written as a
####block becomes a scenario, sinceSCENARIO_HEADERdeliberately matches every level-4 header. I ended up carryingthe link in an HTML comment under the scenario heading instead, which the parser
ignores and rendered Markdown hides.
For anyone who wants the reporting half today, it is
openspec-guard, MIT. The
numbers above, including the ones that make it look bad, are in its
measurements doc.
Happy to fold any of this into a proposal if that is useful when the loop gets
picked up.
ryandemelo commentedon Sep 25, 2026
Thanks @guillaume-flambard, those numbers were really useful. They convinced me the link has to be explicit, so I've been thinking about where it should live. My take is the test file, not the spec.
Something like this in the test:
The anchor is just the capability plus the scenario name, so specs don't need any new syntax. If a scenario name repeats inside a capability you add the requirement to tell them apart. In this repo that only happens 3 times out of 741 scenarios.
A few reasons I prefer the test side:
####line counts as a scenario, which is why Guard had to hide its links in HTML comments.The part I'm most excited about is reading those links through a change's deltas. Something like
openspec coverage --change add-dark-modecould tell you which added scenarios still have no test, which modified scenarios have tests nobody touched on the branch, and which removed scenarios still have tests pointing at them. A plain scenario to test table can't tell you a test is still checking the old behavior, but our deltas can. That comes up a lot here: 32 of the 83 archived changes in this repo modify requirements.It's also a static check, so archive could ask for evidence (#1652) without OpenSpec running commands from a repo file, and
/opsx:verifycould use the report instead of searching the codebase. I'd keep all of it advisory, with an archive gate only for projects that opt in.The one thing I'd want your call on is the tag format, since it gets expensive to change once it's in lots of test files. Does this direction work for you? If it does, I'll write it up as a change proposal before touching any code, and Guillaume, I'd be happy to do that together.
Separately, there's a small fix that helps no matter what:
show --jsondrops requirement and scenario names right now, so there's nothing to link to. That's #1971, with a fix in #1972.alfred-openspec commentedon Sep 25, 2026
Yes, this direction works. Put the durable link on the test side, keep specs free of link metadata, and make the first version static and advisory. A scenario rename should invalidate the reference loudly. That is a useful failure mode, not something to hide with inference.
For the tag, the simple form you proposed is the right base:
@openspec <capability>#<scenario>When a scenario name is ambiguous inside a capability, require the requirement component rather than guessing. The proposal should choose and specify that expanded spelling, plus canonical normalization/escaping, malformed and stale reference behavior, scan boundaries, and whether one test can declare multiple links. I would favor one annotation per link so the grammar stays line-oriented and language-neutral.
Please write the change proposal. Scope it to the reference convention and static resolution/reporting first. Change-aware coverage can build on that. Any archive gate should remain a separate, opt-in policy decision, and this is not a commitment to make test generation part of the default workflow.
ryandemelo commentedon Oct 2, 2026
Written up as #2029. It adds only
openspec/changes/add-test-scenario-references/, scoped to the reference convention and static resolution as you asked, with change-aware coverage, any archive gate and test generation all left out.The six things you asked it to pin down:
<capability>#<requirement>#<scenario>.#separates in both forms so/stays free for a nested capability id (specs-apply.tsalready splits a spec id on/), and the two forms are told apart by counting parts rather than by guessing.-and trimmed, keeping non-ASCII letters rather than transliterating so the convention crosses a language boundary. A segment is valid only when it already equals its own fold, soCli-Showandcli--showare malformed with the canonical spelling named instead of resolving as second spellings. No escaping rule is specified, because the fold alphabet cannot contain#or/and that removes the need for one./*@openspec x*/, is malformed rather than silently trimmed; trimming it means teaching the scanner one comment syntax at a time, and the error message is cheaper.unresolved, naming which part went missing, capability or requirement or scenario, since those are three different repairs. No similarity fallback anywhere in the resolver.git ls-files -co --exclude-standardin a work tree, a declared directory walk otherwise,openspec/and.git/never scanned, oversized and binary files skipped and counted, and the enumeration plus the file counts printed on every run. A coverage report whose scope was wrong would report scenarios as unreferenced when the annotation was simply never read, so the scope is in the output rather than implied.<!-- @openspec ... -->works. No one-line separator, which is also what preserves the no-escaping property above.Numbers from this repository rather than estimates: 747 scenarios across 251 requirements and 36 capabilities, of which 741 resolve from the short form. Six need the requirement component, in three pairs, and after it nothing is ambiguous. A whole-repository scan reads 1,191 files and 9.2 MB in 183 ms, which depends entirely on testing the raw buffer for the literal token before decoding anything.
Resolution reuses
normalizeRequirementName,scenarioNameFromHeaderTextandSCENARIO_HEADERinstead of reading the Markdown a second way, so a reference andshow --jsoncannot end up disagreeing about what a scenario is called.Two things I would rather you pushed back on now than after the grammar is in test files. The annotation is a
(file, line, reference)triple and the tool does not work out which test declaration it sits above, which keeps every language-specific path out of the scanner but means anything writing the token declares a reference. Andcoverageas a name will be confused with line coverage; I used it because you called the follow-on work change-aware coverage and a second word for the same thing seemed worse, buttraceis there if you prefer it.@guillaume-flambard the measurements that ruled out inference are yours, and the design keeps the property you argued for. Two places your experience would help most: whether the git enumeration matches what openspec-guard found it needed in practice, and whether the fold handles the French corpus the way you would expect, since keeping
\p{L}rather than transliterating was a direct response to the 268 of 482 figure.