Skip to content

fix(validation): validate-skills.js accepts a SKILL.md with no body, which is how 66 empty skills shipped #3707

Closed

Description

@eleshar

name: "Bug Report"
about: "Report a reproducible defect"
title: "[Bug]: "
labels: ["type:bug", "area:scripts", "priority:high", "status:needs-triage"]
assignees: []

Describe the bug

scripts/validation/validate-skills.js never asserts that a SKILL.md has a body. It is 36 lines and checks two things: that the folder name is lowercase-kebab, and that a SKILL.md exists at the top level or nested one level down. A file containing nothing but its frontmatter satisfies both and passes.

That is how 66 files shipped empty. skills/*/SKILL.md reached develop holding only YAML frontmatter — 7 to 8 lines each — and were restored by #3680. The restoration fixed the files; the validator that should have caught them was never changed.

To Reproduce

Create a skill whose SKILL.md is frontmatter and nothing else, then run the validator from the directory that holds it:

mkdir -p /path/to/scratch/skills/demo-skill
printf -- '---\nname: demo-skill\ndescription: A skill with frontmatter and nothing else.\n---\n' \
  > /path/to/scratch/skills/demo-skill/SKILL.md

cd /path/to/scratch
node /path/to/repo/scripts/validation/validate-skills.js

Observed:

Skill validation passed.

Exit status 0. A skill that conveys nothing beyond its description is reported as valid.

Expected behavior

validate-skills.js exits non-zero for a SKILL.md that has no body, naming the file — the same way it already names a missing file at line 25.

WordPress Environment

Not applicable. This is a Node.js repository script with no WordPress dependency.

Device and Browser Info

Not applicable. Reproduced on Node.js v24.21.0.

Additional Context

Scope. skills/*/SKILL.md and agents/*/skills/*/SKILL.md. validate-skills.js:9-28 only walks the top-level skills/ directory, so agent-nested skills are not examined at all today.

What "has a body" should mean. Everything after the closing --- of the frontmatter, with at least one non-blank line, excluding the canonical footer block that scripts/agents/includes/header-footer.js appends. dedupe-footers.js --check already identifies those blocks, and #3682 added npm run validate:footers:shape, so the shape tooling exists; this check should reuse it rather than reimplement it.

Why the check was never added. Unknown. The validator has no test file of its own, so a gap in its coverage is invisible to the suite. That is the actual defect behind the 66 empty files: not one mistake, but an untested validator.

Linked Stories/Tasks/PRs

Milestones & Timeline

No deadline. The 66 affected files are fixed; this is the guard that stops the next occurrence.

Definition of Ready (DoR)

  • Defect reproduced against the validator on develop
  • Affected scope identified
  • Related work searched and cited, so this is not a duplicate

Definition of Done (DoD)

  • validate-skills.js exits non-zero for a SKILL.md whose only content is frontmatter, naming the file
  • A SKILL.md whose body is exactly the canonical footer block is reported as having no body
  • A SKILL.md with a body of prose only is reported as valid
  • Every existing skill under skills/ and agents/*/skills/ passes, or each failure is triaged on its own merits
  • A test file exists for validate-skills.js — there is none today, which is how the gap survived
  • The check runs in CI, not only locally

Activity

  1. linear-code commented on Oct 1, 2026

    @linear-code
  2. eleshar commented on Oct 2, 2026

    @eleshar
    ContributorAuthor

    Resolved by #3717, which is merged-pending a code-owner approval. Recorded here so the issue is not re-litigated while that waits.

    Every item in the Definition of Done is now implemented and verified against 673cb9d9:

    DoD item State Evidence
    Exits non-zero for a frontmatter-only SKILL.md, naming the file done [body] finding names file, rule and class; test fails a frontmatter-only skill (#3707)
    A body of exactly the canonical footer reports no body done stripFooter() cuts the whole trailing block, not one line at a time
    A prose-only body reports valid done needed a second fix — see below
    Every existing skill passes or is triaged 754 findings recorded, each <path>#<rule>#<subject> gate reports 754 baselined, 0 new, 0 stale
    A test file exists 135 tests scripts/validation/__tests__/validate-skills.test.js
    The check runs in CI skill-spec job runs npm run validate:skills passed on the head

    Two things worth recording, because both were defects found during review rather than in the original report.

    "A body of prose only is reported valid" was the subtle one. An earlier attempt at the footer check used a flat list of footer openings and matched them in bare text, so ordinary instructions beginning with a generic opener — "Update when the API version changes." — were stripped and reported as an empty body. That fails correct skills, which is worse than the false negative it fixed. It now follows the distinction already drawn in scripts/agents/includes/footer-policy.js: high-confidence openers match bare text, generic ones only match when emphasised. Ten tests fail if that regresses.

    The vocabulary was guessed and had to be corrected. The 16 phrases configured in .github/footers.yml appear nowhere in the repository; the footers actually written are the DEFAULT_FOOTERS. And 35 files carry an emphasised Built by LightSpeedWP… with no brick emoji, which the high-confidence opener cannot match. Both were found by reading the repository rather than the config.

    Scope note: this landed in #3717 rather than as a standalone fix, because #3467 was rewriting the same validator and this issue asks for one validator rather than two. #3717 also covers the agent-nested scope called out here, which the old 36-line script never walked.

    The 754 baselined findings are the remaining conformance work and belong to #3467. This issue is the guard, and the guard is in place.

  3. eleshar commented on Oct 2, 2026

    @eleshar
    ContributorAuthor

    Delivered by #3717, merged into develop as b028d5d. Verified on merged develop: the gate passes (754 baselined, 0 new, 0 stale) and scripts/validation/__tests__/validate-skills.test.js runs 135 tests. The skill-spec CI job now runs npm run validate:skills on every pull request and push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions