Add periodic package hygiene scans #1072

Dockyard can continue rebuilding packages long after their upstream projects have been deprecated, archived, removed, or replaced. Our existing build and security checks do not provide a periodic review of whether we should still distribute each package.

A recent manual review led to #1071, removing seven retired or superseded MCP servers and correcting Octocode's stale image comment, outdated repository URL, and mutable source reference. The corresponding catalog cleanup and hosted replacements are tracked in stacklok/toolhive-catalog#1661.

The same review also showed why findings need interpretation: Notion still accepts changes despite declaring its local server unsupported; Next DevTools declares MIT in package metadata but ships no license text; and Octocode's stale version comment did not mean its published image contained the wrong version.

Proposal

Add a periodic hygiene scan of Dockyard's packaged inventory, with an initial baseline audit and a monthly scheduled run. Keep MCP servers and agent skills as separate scan profiles because their distribution, source pinning, versioning, and replacement models differ.

Cover these areas:

• Upstream lifecycle: archived/deleted repositories, explicit deprecation or end-of-maintenance notices, removed package source directories, yanked/deprecated registry releases, and upstream-recommended replacements. Inspect package-specific sources in monorepos rather than assuming repository health applies to every package.
• Licensing: declared license, presence of license text/notices in the source and distributed package, redistribution eligibility, and license changes. Distinguish missing documentation from an absent license grant; do not infer permission from public source alone.
• Version alignment: reconcile the spec's package/version, published image or skill artifact, registry metadata, source ref, and explanatory comments. Verify artifact contents where practical; do not treat labels or stale comments as proof of the installed version.
• Provenance: repository moves and mismatches, mutable or invalid source refs, changes to publisher/workflow identity, and missing or regressed attestations. Distinguish registry integrity signatures, recorded source references, and cryptographically verified build provenance.
• Maintenance/security practices: dependency update automation, frozen/immutable lockfile use, tests and CI, vulnerability reporting, release practices, and dependency/security scan coverage. Flag stale allowlist explanations that refer to older package/tool versions for review. Inactivity or missing automation should be signals, not automatic removal decisions.
• Replacements/catalog coordination: identify supported hosted services, official maintained packages/images, IDE-integrated replacements, or skills. Check whether hosted replacements already exist in toolhive-catalog and link any required catalog cleanup/addition work.

Reporting and triage

Produce an actionable report with the package/spec path, checked version/ref, timestamp, evidence links, finding category, confidence, and recommended next step. Distinguish confirmed issues, review signals, inaccessible sources, and checks that were not run. Compare with the previous baseline so unchanged findings do not generate duplicate issues or notifications.

Automatically collect deterministic metadata where possible. Any semantic assessment of README notices, licensing ambiguity, or replacement suitability should retain its source evidence and go through maintainer review.

Do not automatically delete packages, change allowlists, waive scanner findings, or classify a package as safe because its build passed. Route findings to a maintainer-owned triage issue or report, with explicit decisions such as retain, fix metadata, request upstream clarification, hold an update, or retire. Record justified exceptions and revisit them when evidence changes.

Acceptance criteria

• Run an initial baseline audit across MCP server and skill inventories, with separate profiles and explicit coverage.
• Add a scheduled monthly hygiene workflow and a manual dispatch option.
• Implement lifecycle, licensing, version alignment, provenance, and maintenance checks, documenting limitations and inaccessible sources.
• Store reports/baselines and deduplicate unchanged findings; notify on new or materially changed actionable findings and scan failures.
• Establish maintainer triage ownership and a documented exception/review process.
• Include catalog coordination and replacement assessment in retirement recommendations.
• Add regression fixtures for the cases exposed by PR Remove retired servers and fix Octocode provenance #1071, including an active repository with a retired subpackage, a renamed repository, a stale image-version comment, and a support disclaimer with continued merges.

AWS skill packaging itself remains tracked in #481; this issue concerns ongoing inventory hygiene.