Make document extraction publicly restorable and publish-ready #85

Goal and agreed boundary

Replace the temporary source-built Microsoft.Extensions dependencies used by the document-extraction providers with official, published NuGet packages, then implement and verify the release path for the Toolkit packages.

“Real packages” means approved upstream package identities available on NuGet.org. The existing source-built packages are genuine compiled packages and useful validation evidence; they are not a supported public distribution channel.

Source integration and publication are separate gates. The validated source in #82 has now merged. This issue tracks the separate public-package/release follow-up; it does not authorize publishing packages, creating releases/tags, or bypassing maintainer review.

Published upstream previews are sufficient. Do not unnecessarily wait for stable releases, and do not guess that all upstream packages have the same version number.

Background / current state

The work is introduced in:

• Add document extraction providers for PdfPig, Azure, Mistral and VisionLanguageModel #82: five independent extraction providers.
• Add local PdfPig extraction with ONNX layout, OCR and table configuration #83: the optional PdfPig/ONNX companion, stacked on the provider branch.

Upstream contracts are merged in dotnet/extensions#7835 (Documents) and dotnet/extensions#7588 (Extraction). A merged upstream PR does not mean the corresponding package version has been published.

As of this handoff, canonical source validation uses:

• Repository: https://github.com/dotnet/extensions.git.
• Pinned commit: 5635a6ed7499739e2411f8288e1bd2b340c2c0e2.
• Tree: 4ea4fe11e7fcc828012824eece4233a5b5f5e591.
• Upstream build SDK: 11.0.100-rc.3.26522.148.
• Temporary core package version: 11.0.0-review.20261106.1.
• Older fork-derived version, also not a public release dependency: 10.10.0-review.20261104.4.

eng\Bootstrap-DocumentExtractionCore.ps1 packs the original four upstream projects without replacing their source/projects. eng\New-DocumentExtractionNuGetConfig.ps1 verifies the bootstrap receipt and routes core restores to that local feed. Current CI transfers these packages between jobs as artifacts. Artifacts expire; they are not a public package feed.

Current third-party restores also use the public dotnet-public mirror, with selected Azure/renderer dependencies mapped to NuGet.org. This is a source-validation setup, not proof that an ordinary customer's NuGet.org-only restore succeeds.

Useful existing acceptance evidence:

• Five-provider canonical source/build/package gate, source head c30c8175d618ad7431d6fe1dc08a76da0090fdb5.
• Companion canonical source/build/package gate, source head 18cfc95bde949bae9eb7551abcd47ac701db8a10.

Both pass on Windows/Linux, including modern and actual portable provider assets on .NET 10. This is a baseline to preserve, not acceptance evidence for future published dependency versions.

Packages in scope

The first five are independently consumable packages. The ONNX companion is separate and must not become a dependency of lightweight PdfPig or the other providers.

All six Toolkit libraries retain net10.0 and netstandard2.0. Supported test/consumer runtime is .NET 10. Do not reintroduce dedicated .NET 8 assets or require a .NET 11 consumer runtime just because the previous upstream bootstrap used SDK 11.

Start here: prerequisites and maintainer decisions

• Work from a branch containing the changes from both provider PRs. If either has not landed, coordinate the dependency/stack rather than duplicate its implementation.
• Read DocumentExtraction\README.md, both extraction workflows, and the files listed below before editing.
• Use the SDK selected by the Toolkit's global.json; use PowerShell 7 for its scripts. Record dotnet --info with validation results.
• With upstream maintainers, identify the published versions containing the required Documents/Extraction APIs. If those versions do not exist yet, mark dependency migration blocked and link the upstream publication follow-up. Do not fabricate versions or publish Microsoft-owned packages yourself.
• With Toolkit maintainers, confirm package ownership/publishing permissions, signing requirements, release versions, tag naming and approval policy.
• Preserve the existing version independence: the five providers and the companion currently have different preview versions. Do not force the companion back to preview.1 or assume all six must always release together.

Suggested release units, to confirm before implementation: a five-provider release group and a separately versioned companion release. A dedicated document-extraction/ tag namespace consistent with existing mevd/ conventions is reasonable, but do not add an active publishing trigger until maintainers approve the exact convention.

Phase 1 — select published upstream dependencies

Create a version/evidence table in the implementation PR for these four identities:

For each selected version:

• Confirm it exists on NuGet.org under the correct official identity. Record the package URL/version and relevant upstream release/source evidence.
• Confirm its API surface contains what the current providers and examples use. Use compile results/API comparison, not only a similar-looking version number.
• Inspect its nuspec/transitive dependencies and .NET 10/portable compatibility. Confirm the entire required dependency closure is publicly restorable.
• Select versions per identity. Do not substitute one common CoreVersion for independently versioned official packages without verifying that they really align.
• Re-evaluate System.Text.Json requirements. The current upstream snapshot needs JSON 11 RC3, including for manually selected portable provider DLLs. Do not retain an obsolete RC pin unnecessarily, introduce a downgrade, or let the host's JSON 10 silently mask a required package dependency.

If an upstream package is missing or incompatible, stop that phase with concrete evidence. Do not silently fall back to source-built packages, a warmed cache, a private feed, or copied DLLs.

Phase 2 — migrate active references and ordinary restore/CI

Files to inspect/change

• Directory.Packages.props: extraction package versions and extraction-scoped AI/JSON overrides.
• DocumentExtraction\src\Directory.Build.props and DocumentExtraction\test\Directory.Build.props.
• eng\DocumentExtraction.PackageSmoke\DocumentExtraction.PackageSmoke.csproj and Program.cs.
• DocumentExtraction\samples\PdfPigOnnx.PackageSmoke\PdfPigOnnx.PackageSmoke.csproj and its consumer code.
• eng\Test-DocumentExtractionPackages.ps1.
• eng\New-DocumentExtractionNuGetConfig.ps1 and eng\Bootstrap-DocumentExtractionCore.ps1.
• .github\workflows\document-extraction.yml and .github\workflows\pdfpig-onnx.yml.
• Component/provider READMEs, packed skills, and any active install/build examples containing temporary versions.

Tasks:

• Replace active review-version references with selected published versions, including explicit middleware Version/VersionOverride values in both smoke projects.
• Keep DocumentExtractionBuild-scoped overrides scoped. Do not accidentally upgrade MEVD's AI/JSON dependencies as a side effect.
• Make the default development/CI restore path consume published dependencies using the normal public configuration. Remove the source-bootstrap prerequisite/job/core-artifact transfer from that path.
• Refactor package acceptance to work in published-dependency mode. It currently requires -CorePackageDirectory, reads core-bootstrap.json, and hard-codes the source commit/SDK/review version. Removing the argument at a call site alone is not sufficient.
• Replace source-bootstrap guards with published-package checks: selected package identities/versions, fresh resolved closure, actual provider/core DLL bindings, and provenance appropriate to the published packages.
• Derive each published core's compatible selected asset from restore metadata. Official packages may have different versions/TFM sets than the source snapshot; do not require all four to contain lib/net10.0 or all six historical upstream TFMs.
• Make both smoke consumers accept the selected Toolkit candidate versions rather than always consuming preview.1/preview.2. Verify restored versions against the actual candidate nuspecs.
• Preserve explicit selection/execution of the actual netstandard2.0 provider DLLs on .NET 10. A normal .NET 10 restore selecting only modern provider DLLs is not portable acceptance.
• Handle a plain public nuget.config that has no existing packageSourceMapping element. If the harness creates a candidate feed/mapping, create the required XML safely and preserve the public source's mapping.
• Use exactly one key per source URI. The canonical migration already exposed a NuGet URI-deduplication failure when different keys used the same NuGet.org URI; do not reintroduce it.
• Retain bootstrap tooling only as clearly opt-in diagnostic/reproduction tooling if maintainers want it. It must not be invoked by normal restore, default CI, publishing, or package installation. Otherwise remove obsolete tooling/callers together.
• Search for remaining active review-version/bootstrap assumptions. Historical notes and deliberately negative test fixtures may retain them; production, ordinary CI, and release paths may not.

CommunityToolkit.AI.slnx already includes the Toolkit source/tests, with the companion/sample added in PR83. Verify paths still exist, but do not add upstream source projects to the Toolkit solution and do not hide Toolkit projects merely to make restore appear successful.

Phase 3 — prove public dependency closure and actual package consumption

Run on Windows and Ubuntu with new output/package-cache directories, not a developer's warmed global package cache.

Required matrix

• Build all six Toolkit libraries for both shipped TFMs.
• Run the five-provider suites on .NET 10 using modern provider assets.
• Run those suites on .NET 10 explicitly using actual netstandard2.0 provider assets.
• Run the companion suites in both profiles and the offline provisioning helper tests.
• Pack the five providers and companion into clean candidate directories, preserving the intended per-package versions.
• Restore actual candidate packages with only a local Toolkit candidate feed plus NuGet.org. No local/source-built Microsoft.Extensions feed, core-bootstrap.json, Azure Artifacts mirror, private source or source-project substitution may be needed for this proof.
• Inspect project.assets.json, restored package source metadata, selected versions and loaded DLL paths/hashes. Confirm no review core packages, old binaries, or source project references satisfied the consumer.
• Fail compilation without CTDE0001 opt-in, then compile/run with deliberate opt-in. Do not count an unrelated compiler error as the expected experimental failure.
• Verify internal legacy attribute visibility, required/init metadata and framework identity as the existing harness does.
• Compile the exact packed README/skill examples. Exercise the optional middleware/logging example with the published middleware package, rather than treating a validation-only DLL-load probe as proof that middleware behavior works.
• Run native companion acceptance with its explicitly selected CPU runtime and synthetic PDF/ONNX fixtures. Keep real-weight tests explicitly skipped unless separately authorized; do not relabel a skip as accuracy coverage.

Starting commands (PowerShell, repository root)

These build/test commands should work after the published-reference migration. Use the checked-in public nuget.config, not a bootstrap-generated core config:

$proofRoot = Join-Path ([IO.Path]::GetTempPath()) ("ctk-extraction-" + [guid]::NewGuid().ToString('N'))
New-Item -ItemType Directory -Path $proofRoot | Out-Null
$env:NUGET_PACKAGES = Join-Path $proofRoot 'packages'
$env:NUGET_HTTP_CACHE_PATH = Join-Path $proofRoot 'http-cache'
$candidateDirectory = Join-Path $proofRoot 'candidates'
$newConsumerOutputDirectory = Join-Path $proofRoot 'five-provider-consumers'
$newCompanionOutputDirectory = Join-Path $proofRoot 'companion-consumers'

dotnet restore .\DocumentExtraction\DocumentExtraction.slnf --configfile .\nuget.config
dotnet build .\DocumentExtraction\DocumentExtraction.slnf -c Release --no-restore
dotnet test .\DocumentExtraction\DocumentExtraction.slnf -c Release --no-build --no-restore
dotnet restore .\DocumentExtraction\DocumentExtraction.slnf --configfile .\nuget.config -p:DocumentExtractionTestAsset=netstandard2.0
dotnet test .\DocumentExtraction\DocumentExtraction.slnf -c Release --no-restore -p:DocumentExtractionTestAsset=netstandard2.0

dotnet restore .\DocumentExtraction\PdfPig.OnnxLayoutAnalysis.slnf --configfile .\nuget.config
dotnet build .\DocumentExtraction\PdfPig.OnnxLayoutAnalysis.slnf -c Release --no-restore
dotnet test .\DocumentExtraction\test\PdfPig.OnnxLayoutAnalysis.UnitTests -c Release --no-build --no-restore
dotnet restore .\DocumentExtraction\test\PdfPig.OnnxLayoutAnalysis.UnitTests --configfile .\nuget.config -p:DocumentExtractionTestAsset=netstandard2.0
dotnet test .\DocumentExtraction\test\PdfPig.OnnxLayoutAnalysis.UnitTests -c Release --no-restore -p:DocumentExtractionTestAsset=netstandard2.0
.\DocumentExtraction\src\PdfPig.OnnxLayoutAnalysis\Test-ModelProvisioning.ps1

dotnet pack .\DocumentExtraction\DocumentExtraction.slnf -c Release --output $candidateDirectory
dotnet pack .\DocumentExtraction\src\PdfPig.OnnxLayoutAnalysis\PdfPig.OnnxLayoutAnalysis.csproj -c Release --output $candidateDirectory

The portable restore and test must use the same DocumentExtractionTestAsset=netstandard2.0 property. Do not reuse a modern-only assets file. Pack is allowed to rebuild/restore using the root public configuration so previous profile-specific builds cannot leave an incomplete package.

These are interactive starting commands. In an automation script, check $LASTEXITCODE after every external command and stop on failure; a later successful command must not hide a failed restore/test. Run the equivalent commands in both CI operating-system jobs.

After refactoring the acceptance script for published dependencies, its invocation should no longer require a core directory; for example:

.\eng\Test-DocumentExtractionPackages.ps1 `
  -PackageDirectory $candidateDirectory `
  -NuGetConfig .\nuget.config `
  -OutputDirectory $newConsumerOutputDirectory

.\eng\Test-DocumentExtractionPackages.ps1 `
  -PackageDirectory $candidateDirectory `
  -NuGetConfig .\nuget.config `
  -OutputDirectory $newCompanionOutputDirectory `
  -ConsumerProject .\DocumentExtraction\samples\PdfPigOnnx.PackageSmoke\PdfPigOnnx.PackageSmoke.csproj

Each consumer output directory is new; the harness must isolate its own package/HTTP cache. Preserve the companion version override when validating a differently versioned candidate. The existing harness can be extended rather than replaced; add regression tests for its published mode.

Archive inspection

• Exact intended package IDs/versions, only lib/net10.0 and lib/netstandard2.0 provider folders, and corresponding nuspec dependency groups.
• No review/bootstrap dependency versions in production nuspecs; no bundled upstream DLL copies or core nupkgs used to bypass dependency resolution.
• Correct license/readme/SourceLink/repository metadata and documented symbol policy.
• CTDE0001 remains; upstream MEDE0001 / EXTEXP0019 opt-ins remain distinct where the published APIs require them.
• All six package-scoped skills are present with valid frontmatter, self-contained relative links and compilable examples.
• Companion includes its helper, manifest, notices and four skill files; no model weights, automatic acquisition hooks or unintended native runtime policy change.
• Lightweight packages still do not pull in the ONNX companion/rendering/model stack.

Phase 4 — implement a safe Toolkit release path

Read these existing examples first:

• .github\workflows\mevd.yml: existing tag parsing, component-specific packing and release caller.
• .github\workflows\release.yml: reusable signing/GitHub Release/NuGet publication workflow.
• .github\workflows\SignClientFileList.txt: currently **/CommunityToolkit.*.

The extraction workflows currently upload validation artifacts only; they do not call the publishing workflow. A merge to main must continue to perform validation, not publication.

Tasks:

• Implement the maintainer-approved extraction release-unit/tag selection using an explicit whitelist of project/package IDs. Unknown or malformed selectors must fail; do not turn arbitrary tag text into a project path.
• Pack only the selected extraction release unit. Do not pack the whole root solution for an extraction release and accidentally include MEVD or upstream core packages.
• Choose one authoritative release package set from the selected build job. Windows/Linux both validate it; their separate source-build outputs must not be merged into a publishing artifact containing duplicate package identities/versions.
• Require approved published dependencies before a release build. Add a guard rejecting active temporary core review versions and bootstrap/source-feed dependencies.
• Add a dry-run/validation-only path that packs, inspects and consumes candidates but cannot create tags/releases, sign with production credentials or push to NuGet.org. Default manual validation should be non-publishing.
• Reuse the signing/publishing workflow where practical. It currently expects nuget-packages and nuget-list artifacts; align the new caller or deliberately parameterize that contract.
• Verify the signing file list actually covers all intended extraction assets and no Microsoft-owned/unselected packages. Determine which NuGet and/or contained-binary signatures the repository policy requires; inspect the result rather than assuming a successful tool install means packages are signed.
• Verify final signed candidates and publish those same artifacts. Do not rebuild/repack between final acceptance and publication; record hashes after signing because signing can change bytes.
• Confirm NuGet package-prefix ownership, existing signing secrets, OIDC NUGET_USER, least-privilege permissions and the nuget-release-gate environment/approval policy with maintainers. Do not put secret values in code, logs or this issue.
• Ensure all public side effects, including GitHub Release creation, follow the approved release gate. Existing reusable release creation is a separate job from the NuGet environment gate; a dry run must not accidentally call it.
• Do not expose production credentials to pull requests/fork code or unvalidated manual branches.
• Add a release-unit package whitelist check immediately before upload/sign/push: expected count and identities, nonempty artifacts, no duplicates, no unselected Toolkit packages, no core Microsoft.Extensions packages or weights.
• Handle immutable versions explicitly. --skip-duplicate is not proof that a previously published package contains the current build. Verify an existing version or choose a new approved version; never imply a different package was overwritten.

Phase 5 — docs, promotion and post-publication checks

• Update component/provider READMEs and packed skills so install examples refer to actually published provider/upstream identities. Explain native runtime/model setup for the companion and backend requirements for Azure/Mistral/VisionLanguageModel.
• Remove active claims that ordinary users need the canonical source bootstrap, SDK 11 to build upstream, or the temporary review feed. Preserve useful historical validation notes with explicit historical labels.
• Document contributor build steps, candidate validation, tag/release-unit selection and maintainer-only promotion steps in the repository.
• Attach the version matrix, fresh restore/build/test logs, actual package/loaded-asset manifest and dry-run evidence to the implementation PR. Distinguish source HEAD from a CI-tested merge commit; do not relabel artifact provenance.
• Obtain maintainer approval for the first release. This issue does not grant that approval.
• After an approved publication, use a separate empty cache and NuGet.org-only configuration to install/restore the released packages, compile/run offline smoke consumers and check skill delivery. Account for bounded feed-index propagation; failed restore remains a failure, not a successful release claim.
• Record published package URLs/versions. When comparing uploaded/downloaded packages, account for NuGet repository signing potentially changing whole-archive bytes; verify the package identity, payload/provenance and applicable signatures, not a misleading raw ZIP hash comparison.

Acceptance / definition of done

Implementation is publish-ready when

• Every required upstream dependency has a compatible, officially published NuGet.org version, with recorded evidence and no review-version production references.
• Fresh public-dependency restores/builds pass; ordinary development and release validation no longer require the source bootstrap or a private/mirror/core feed.
• All six actual candidate packages pass modern/portable-on-.NET-10 acceptance on Windows/Linux, including compiler opt-in, native companion fixtures, packed examples and skill/archive checks.
• Default source integration/CI remains non-publishing; release selection, signing/artifact contracts, permissions, approval gates and non-publishing dry run are implemented and reviewed.
• Unrelated MEVD dependency/release behavior is unchanged; current provider APIs/names/defaults and the two-TFM support policy are preserved.
• Missing official versions, wrong assets, stale caches, unexpected packages and invalid release selectors have negative regression coverage and fail visibly.

Publication follow-through

• Maintainers either perform the authorized first release and record NuGet.org-only post-publication smoke results, or explicitly split/link a maintainer-owned promotion issue. Do not close the overall public-consumption work with only source-built CI evidence.

Non-goals / safety boundaries

No API renaming/redesign, model-quality certification, new runtime targets, GPU certification, model acquisition, paid service calls, publishing Microsoft-owned review packages, or incidental MEVD cleanup. Preserve existing evidence and known recognition/table-quality limitations. No silent fallback, TLS/certificate-validation bypass, restored-cache substitution or automatic model downloads to make a gate look green.