validate: MODIFIED/REMOVED/RENAMED-from headers that don't exist in base spec aren't caught until archive (proposal: opt-in cross-change MODIFIED) #1112

Activity

1. clay-good

   added 5 commits that reference this issue on Jun 29, 2026

   docs(proposal): add prevent-silent-spec-drop change

   ...

   23f3c10

   docs(proposal): add prevent-silent-spec-drop change

   ...

   48ea8ae

   docs(proposal): add prevent-silent-spec-drop change

   ...

   af48253

   docs(proposal): add prevent-silent-spec-drop change

   ...

   3e1e8fb

   docs(proposal): add prevent-silent-spec-drop change

   ...

   04a386d
2. clay-good

   added

   bugSomething isn't workingSomething isn't working

   on Jun 29, 2026
3. clay-good

   added 3 commits that reference this issue on Jun 29, 2026

   docs(proposal): add prevent-silent-spec-drop change

   ...

   6d3298d

   docs(proposal): add prevent-silent-spec-drop change

   ...

   9ae0781

   docs(proposal): add prevent-silent-spec-drop change

   ...

   dd24830
4. ryandemelo commented on Aug 19, 2026

   

   ryandemelo

   on Aug 19, 2026

   Contributor

   More actions

   Case 2 here (Change B MODIFYing a requirement Change A introduced but hasn't archived) is the same shape I ran into from the other side, working on cross-change overlap reporting for #1669 — so a note in case it's useful for scoping the fix.

   The two implementations that disagree are validator.ts:607 (if (!current) continue; — a MODIFIED whose target is missing validates clean) and specs-apply.ts:454 (MODIFIED failed ... - not found). Anything else that needs to answer "does this delta apply?" — overlap reporting, a --dry-run, the diff command in #427 — becomes a third implementation of the same question unless the check lands as one function both validate and buildUpdatedSpec call.

   I tried to answer it a second time and got it wrong twice, verified against real archive runs: specs-apply.ts has several branches that treat a repeat as already-synced rather than as a collision, and RENAMED's version of that tests the counterpart header's presence, so a check reasoning about one header at a time gets renames wrong in both directions. I've written that up in more detail on #1113, which looks like the right home for the fix.

   Not proposing anything competing here — #1113 is open for this and @ixxie has offered to finish it.
5. ixxie commented on Aug 19, 2026

   

   ixxie

   on Aug 19, 2026

   Last edited by ixxie

   More actions

   @ryandemelo you may wanna have a look at #1683 and the associated PR; I came to the design trying to solve CI issues but it should make #1669 a lot simpler, at least in the long term.

   Of course since my proposal is an opt-in mode with migrations, I guess it would be a bit complicated during the intermediate dual support time. But Im convinced explicit state makes everything easier.
6. ixxie

   added 2 commits that reference this issue on Aug 19, 2026

   feat(lifecycle): experimental `lifecycle: status` mode — state as data

   ...

   3b11baa

   feat(lifecycle): experimental `lifecycle: status` mode — state as data

   ...

   d3a4361
7. clay-good

   added

   design-reviewNeeds product/design decisionNeeds product/design decision

   on Aug 27, 2026
8. pull

   added a commit that references this issue on Sep 2, 2026

   feat(validate): report the deltas archive would refuse (Fission-AI#1710)

   ...

   a4fcdbe
9. ryandemelo commented on Sep 7, 2026

   

   ryandemelo

   on Sep 7, 2026

   Contributor

   More actions

   Thanks @ixxie — read it properly, and replied over on #1683 since the argument is with II and IV rather than with anything on this issue. Short version for anyone following this thread: I think IV is the decision that bears on #1112, and it states the shared-implementation property for sync --check and the fold, which is a different pair from the validate / buildUpdatedSpec split this issue is about. Same principle, neither one closes the other.
10. jean-sturacci commented on Oct 1, 2026

    

    jean-sturacci

    on Oct 1, 2026

    More actions

    We hit both of your cases on a 73-change cross-repo project and ended up building an external deny-gate for them. Since #1918 was closed into this issue with the remaining question stated as "making validate fail on a lone typo without blocking valid stacked changes", here is how we discriminate the two — in case the shape is useful, not as a request.

    The discriminator we use: resolve the absent target against the declared predecessor's ADDED requirements.

    Our changes carry a blocked_by: list (a project-local field — OpenSpec has no equivalent, which is the obvious limit of this report). The gate, run on archive:

    1. Parse the ## MODIFIED headers out of the change's deltas.
    2. For each, look for an exact match in openspec/specs/<cap>/spec.md.
    3. On a miss, look for the same header under ## ADDED Requirements in each declared predecessor's delta.
    4. Found there → resolved, not a finding. That is your Case 2, correctly queued, and firing on it would report correctly-ordered work as broken.
    5. Not found anywhere → deny, naming the header and the spec file. That is your Case 1.

    So the signal that separates them is not anything about the header — it is whether the author declared the dependency. A typo declares nothing.

    Two things we got wrong first, both worth more than the mechanism:

    🔴 Our first version took 27.2 seconds. A PreToolUse hook that exceeds the harness timeout fails silently, so for a deny gate that means the archive proceeds unguarded — the exact false all-clear it existed to remove. If you do put this in validate, the in-memory merge is already paid for by #1710, but the predecessor walk is not. Ours now measures 0.8–3.9s over 168 archived changes, and we plan against the worst case, never the best.

    🔴 Matching must be anchored. An archived 2026-08-11-<id> must not satisfy a blocked_by of a different id that happens to be a suffix of it.

    A second gate, because the first answers the wrong question for a reader. The deny gate judges only the change being archived. We also compute, across the whole in-flight set, which other changes declare a requirement this delta also declares under MODIFIED/REMOVED/RENAMED — the three body-reproducing sections. That one is an ask, never a deny: two in-flight changes touching one requirement is legitimate and, with most of ours sharing a capability, routine. The resolution is ordering, not exclusion, and a gate that blocks the normal path is one people route around.

    Both reuse the same heading extractor and the same scope expression byte-for-byte, because two mechanisms parsing the same text differently is worse than one — both keep running and only one is right.

    Why we think INFO alone was not enough for us, stated as experience rather than as a disagreement with the call made in #1918: a MODIFIED requirement reproduces its whole body, so when two changes modify one requirement and the second archives against the pre-rebase body, archive reports success while reverting everything the first added. We measured one such pair where the scenario set of the second was a strict superset of the current spec's — it added one scenario and dropped none — so the 1.8.0 scenario-drop guard correctly did not fire. The loss would have been confined to prose inside scenarios both deltas kept, which nothing compares. That case is invisible to any check keyed on scenario names.

    Happy to answer questions; we are not asking for the blocked_by field itself, only offering the discrimination rule in case it saves someone the two wrong turns above.
11. ryandemelo commented on Oct 2, 2026

    

    ryandemelo

    on Oct 2, 2026

    Contributor

    More actions

    @jean-sturacci this is the most useful thing on this issue. Rather than agree with it I went and checked the part that bears on my own work against current main, then measured it here, so the below is confirmation and one correction to my own first attempt at measuring it.

    The guard is name-keyed, exactly as you say.

    Two call sites, the archive-time one at specs-apply.ts:542 and the validate-time one at validator.ts:765. Both call diffScenarioNames, whose first two lines are:

    const currentNames = parseScenarioBlocks(current.raw).map((scenario) => scenario.name);
    const incomingNames = parseScenarioBlocks(incoming.raw).map((scenario) => scenario.name);

    Names only, and nothing anywhere in src/core/ compares one scenario body against another. The apply path then does nameToBlock.set(key, mod) at specs-apply.ts:555, replacing the whole block. The only thing between the guard and that line is a normalizeBlockRaw comparison at 552, and it exists to decide whether a write happened, not to warn that one did. So your case lands precisely where you said: scenario set a superset, guard silent, requirement statement and every kept scenario's body replaced wholesale.

    How often the shape occurs in this repository.

    85 archived changes, 122 delta spec files, 93 MODIFIED blocks, 52 distinct (capability, requirement) pairs modified.

    • 14 of those 52 requirements were modified by more than one archived change, with a long tail: one claimed by 10 changes, one by 9, one by 7.
    • 41 consecutive claim pairs on one requirement, 29 of them within 14 days of each other.
    • 15 archive commits where more than one change modified the same requirement with differing block text, so one won and the rest were discarded with nothing reported.
    • Of those 15, 7 also disagreed on the scenario name set, which the guard can see. 12 disagreed on the body of a scenario that all of them kept, which the guard cannot.

    One named case, since an aggregate is easy to wave away. Four changes archived together on 2025-10-14 all modified cli-init :: Slash Command Configuration, and all four keep the scenario "Generating slash commands for OpenCode". Two of them disagree on the path inside it, .opencode/command/ against .opencode/commands/. One of those is what got written and nothing said a choice had been made.

    What I did not find, which matters as much.

    I also wrote the stronger test, because "the bodies differ" is not the same claim as "content was lost". For each archived MODIFIED block: reconstruct the main spec as it stood at the parent of that change's archive commit, walk the full git history of that requirement's statement text, and flag any delta whose statement matches a value that appeared earlier in that history than the base did. That is a revert to a stale base, as opposed to an author shortening something on purpose.

    Zero hits across 84 blocks.

    My first pass was the easy measurement, "did kept prose get shorter", and it reported 14 cases. Every one I opened was either a batch artifact, where I had compared three deltas from a single archive commit against the same parent, or a deliberate rewrite. That is the number I would have posted had I stopped there, and it would have been wrong.

    I read the zero as supporting your framing rather than weakening it. Whether a body difference is intent or an accident is not decidable from the artifacts. The tool cannot tell, and it never asks. That is the same conclusion I reached from the other direction measuring scenario omissions: the omission itself carries no signal about whether the author meant it, so the marker has to be authored.

    On blocked_by: the field exists as a proposal.

    True of shipped OpenSpec, not of the queue. The open add-change-stacking-awareness change proposes dependsOn as change metadata, with dependsOn as the source of truth for archive ordering, alongside provides, requires, touches and parent. Your discriminator plugs into that field directly, and your report is the strongest argument for it so far because it is a measured account of what goes wrong without it. Worth saying so there if you have the appetite.

    Your anchored-matching warning applies to it immediately: dependsOn will hold change ids, archived ids carry a date prefix, and a suffix match would let 2026-08-11-foo satisfy a dependency on a different id ending in foo.

    Three checks, not one.

    Worth separating, because this issue now carries all three and they have different answers.

    1. MODIFIED target absent from the base. This issue. Decidable from one change plus the main specs.
    2. Two in-flight changes both claim one requirement. feat(validate): report requirements two active changes both claim #1698, design in docs(openspec): propose reporting requirements two active changes both claim #1963. Advisory, reported without ranking, which is the same ask-never-deny conclusion you reached independently. That was the part of the design I was least sure of, so I will take the corroboration.
    3. Target present, but the body the delta reproduces is stale. Yours. Neither 1 nor 2 catches it. 2 reports the precondition at validate time, which is the earliest anything can, but it warns and does not compare the two bodies, so an author who ignores the warning still gets last-writer-wins.

    I am not proposing 3 here. It needs a body comparison that does not exist anywhere in the codebase, and after the 27.2 second story I would rather see the cost measured before anyone designs the check than after.

    On the timeout, taking the general lesson.

    The asymmetry is the part worth keeping: a report that is slow is just slow, a gate that is slow is a false all-clear. It changed how I wrote up a static scan I have been working on elsewhere in the tool. The cheap prefilter is now a fixed decision with the measured worst case beside it, rather than an implementation detail someone later has to rediscover the reason for.
12. fidelix commented on Oct 6, 2026

    

    fidelix

    on Oct 6, 2026

    More actions

    Capability-path form of the same split. Main has scheduling/dispatch. The change puts ## MODIFIED Requirements for Assign a technician under specs/scheduling/dispatchh/spec.md (the directory name is mistyped).

    openspec validate add-region-filter --strict exits 0. The only finding is this INFO line:

    ℹ [INFO] scheduling/dispatchh/spec.md: Archive would refuse this delta: scheduling/dispatchh: target spec does not exist; only ADDED requirements are allowed for new specs. MODIFIED and RENAMED operations require an existing spec.
    

    openspec archive add-region-filter --yes exits 1:

    scheduling/dispatchh: target spec does not exist; only ADDED requirements are allowed for new specs. MODIFIED and RENAMED operations require an existing spec.
    Aborted. No files were changed.
    

    Expected: validate --strict should fail with an error when a MODIFIED or REMOVED delta targets a capability or requirement that is not in main, consistent with archive. This is the lone typo left open when #1918 was closed into this issue: the capability directory does not exist, rather than a requirement header missing inside an existing spec.

    Environment: OpenSpec 1.14.1 (npx -y @fission-ai/openspec@1.14.1 --no-color), Node v24.21.0, macOS (Darwin 25.6.0, arm64). Same INFO line and same exit codes on 1.12.0. I ran the script twice from scratch on each version. Both runs on a version matched.

    Impact: Two branches editing the same capability in parallel is common. validate --strict is the check that runs on the branch, and it stays green for a mistyped capability path. Archive is the first command that refuses the delta.

    Repro

    #!/usr/bin/env bash
    # Bug: validate --strict accepts a MODIFIED delta whose capability does not
    # exist. archive refuses the same delta.
    set -euo pipefail
    
    OPENSPEC=(npx -y @fission-ai/openspec@1.14.1 --no-color)
    ROOT="$(mktemp -d)"
    cd "$ROOT"
    git init -q
    echo "+ ${OPENSPEC[*]} init --tools none --no-animation ."
    "${OPENSPEC[@]}" init --tools none --no-animation . >/dev/null
    echo "(exit 0)"
    
    mkdir -p openspec/changes/add-dispatch/specs/scheduling/dispatch
    cat > openspec/changes/add-dispatch/.openspec.yaml <<'EOF'
    schema: spec-driven
    created: 2026-10-06
    EOF
    cat > openspec/changes/add-dispatch/specs/scheduling/dispatch/spec.md <<'EOF'
    ## Purpose
    
    Assigns one technician to a job that is ready to be dispatched.
    
    ## ADDED Requirements
    
    ### Requirement: Assign a technician
    The system SHALL assign a technician to a ready job.
    
    #### Scenario: Single technician
    - **WHEN** a job is ready for dispatch
    - **THEN** the system assigns one technician
    EOF
    
    echo "=== setup: archive the base capability ==="
    echo "+ ${OPENSPEC[*]} validate add-dispatch --strict"
    "${OPENSPEC[@]}" validate add-dispatch --strict
    echo "(exit 0)"
    echo "+ ${OPENSPEC[*]} archive add-dispatch --yes"
    "${OPENSPEC[@]}" archive add-dispatch --yes
    echo "(exit 0)"
    
    echo "=== main specs ==="
    find openspec/specs -type f -print | sort
    
    echo "=== change add-region-filter targets scheduling/dispatchh (typo) ==="
    mkdir -p openspec/changes/add-region-filter/specs/scheduling/dispatchh
    cat > openspec/changes/add-region-filter/.openspec.yaml <<'EOF'
    schema: spec-driven
    created: 2026-10-06
    EOF
    cat > openspec/changes/add-region-filter/specs/scheduling/dispatchh/spec.md <<'EOF'
    ## MODIFIED Requirements
    
    ### Requirement: Assign a technician
    The system SHALL assign a technician to a ready job only within the job's service region.
    
    #### Scenario: Single technician
    - **WHEN** a job is ready for dispatch
    - **THEN** the system assigns one technician
    EOF
    
    echo "=== validate --strict ==="
    echo "+ ${OPENSPEC[*]} validate add-region-filter --strict"
    set +e
    "${OPENSPEC[@]}" validate add-region-filter --strict
    validate_status=$?
    set -e
    echo "(exit ${validate_status})"
    
    echo "=== archive ==="
    echo "+ ${OPENSPEC[*]} archive add-region-filter --yes"
    set +e
    "${OPENSPEC[@]}" archive add-region-filter --yes
    archive_status=$?
    set -e
    echo "(exit ${archive_status})"
    
    echo "=== proof ==="
    if [[ "$validate_status" -ne 0 ]]; then
      echo "BUG NOT REPRODUCED: validate --strict exited ${validate_status}"
      exit 1
    fi
    if [[ "$archive_status" -eq 0 ]]; then
      echo "BUG NOT REPRODUCED: archive exited 0"
      exit 1
    fi
    if [[ -e openspec/specs/scheduling/dispatchh/spec.md ]]; then
      echo "BUG NOT REPRODUCED: archive created the typo capability"
      exit 1
    fi
    echo "BUG REPRODUCED: validate --strict exited 0 and archive exited ${archive_status}."
    echo "Main spec left in place:"
    find openspec/specs -type f -print | sort

    Actual output (1.14.1, trimmed)

    archive add-dispatch --yes exits 0. Main specs are openspec/specs/.gitkeep and openspec/specs/scheduling/dispatch/spec.md. Then:

    === validate --strict ===
    + npx -y @fission-ai/openspec@1.14.1 --no-color validate add-region-filter --strict
    Change 'add-region-filter' is valid
    ℹ [INFO] scheduling/dispatchh/spec.md: Archive would refuse this delta: scheduling/dispatchh: target spec does not exist; only ADDED requirements are allowed for new specs. MODIFIED and RENAMED operations require an existing spec.
    (exit 0)
    === archive ===
    + npx -y @fission-ai/openspec@1.14.1 --no-color archive add-region-filter --yes
    Task status: No tasks
    
    Specs to update:
      scheduling/dispatchh: create
    scheduling/dispatchh: target spec does not exist; only ADDED requirements are allowed for new specs. MODIFIED and RENAMED operations require an existing spec.
    Aborted. No files were changed.
    (exit 1)
    === proof ===
    BUG REPRODUCED: validate --strict exited 0 and archive exited 1.
    Main spec left in place:
    openspec/specs/.gitkeep
    openspec/specs/scheduling/dispatch/spec.md