Skip to content

feat(integration): make the integration config a variant table with per-product pages - #528

Merged
edwinyjlim merged 5 commits into
mainfrom
dag/c2-variant-table
Nov 16, 2026
Merged

edwinyjlim merged 5 commits into
mainfrom
dag/c2-variant-table

Conversation

@edwinyjlim

@edwinyjlim edwinyjlim commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

context/skills/integration/config.yaml becomes the variant table. Each row names the variant it is written for under match, and lists the error tracking and feature flag pages for that variant under docs. The build writes those pages into the row's pack, lists them in a variant.json at the pack's root, and publishes the variant as a menu tag.

Every pack's SKILL.md is byte-identical, and the table only adds files and tags. One fix changes skills published today: two pages of a skill that share a file name are now both written, where the second replaced the first.

Stack

The second half of the migration, behind wizard-agent-dag-skills. The first half, behind wizard-agent-dag, is in context-mill 1.65.0 (#447) and wizard 2.94.0 (PostHog/wizard#1551 to PostHog/wizard#1554).

PR Repo What it does How it fits in
A · Variant table · #528 (this PR) context-mill Each integration variant lists every product's pages, and its pack ships them with a variant.json. On main. Adds files and tags, and writes both pages where one replaced another.
B · Step skills · #529 context-mill Publishes each orchestrator step skill once, as integration-<step>. On A. Adds skills only, and the skills they replace keep publishing.
C · Freeze note · #532 context-mill Says the agent prompts are frozen and edited in the wizard. On main, on its own. Documentation only. Merges when wizard-agent-dag reaches 100%.
PR 5 · Read repackaged skills · PostHog/wizard#1559 wizard With wizard-agent-dag-skills on, a task reads the step skill from B, completed from the pack from A. On main. Does nothing while the flag is off.

A and B are one GitHub stack and merge together. Apart from the page fix in A, they only add files, skills and tags, and no run reads those until the flag is on.

Rollout

wizard-agent-dag-skills is a percentage flag, independent of wizard-agent-dag. Turning it off returns every run to the skills it reads today, and a run pinned to a context-mill release reads those throughout.

Step What changes for a run What is checked
1. A and B merge, and context-mill releases On both sides of the flag, the error tracking step gains the installation page it was missing on Expo, FastAPI, SvelteKit, Swift and WordPress. Nothing else: the packs gain files and the menu gains entries that no run reads yet check:bundle-diff --additive and check:repackaged-skills on the release build
2. PR 5 merges, and the wizard releases with the flag off Nothing A run with the flag off matches the previous release
3. The flag is on for PostHog staff Staff runs read the repackaged skills pnpm verify:repackaged-skills against the release, and a workbench run with the flag on compared with one with it off.
4. The flag is at 50%, for about a day Half of runs read the repackaged skills The two sides by the orchestrator_skills tag: run success, task failures, and any orchestrator repackaged skill unavailable
5. The flag is at 100% Every run reads the repackaged skills The same signals, for about a week
6. Cleanup Nothing A later PR deletes the flag and the old lookup. Context-mill stops publishing the replaced skills once older wizard versions have drained

C merges on its own, when wizard-agent-dag reaches 100%.

What a row gains

- id: nextjs-pages-router
  framework: nextjs
  docs_urls:
    - https://posthog.com/docs/libraries/next-js.md
  match: { framework: nextjs, frameworkVariant: pages-router }
  docs:
    error-tracking:
      - https://posthog.com/docs/error-tracking/installation/nextjs.md
    feature-flags:
      - https://posthog.com/docs/feature-flags/installation/react.md
      - https://posthog.com/docs/libraries/next-js.md
      - https://posthog.com/docs/feature-flags/installation/nodejs.md
Key Meaning
match The trait values the pack is written for. A key is a trait name as the wizard spells it. frameworkVariant is the id the wizard's framework config reports.
docs Each product's pages for the variant. docs_urls stays the integration pages.
gaps The products a row has no pages for.
products, on the group The products every row lists or names as a gap. The build fails on a row that does neither.

A row lists its own pages for a product: the ones under docs_urls in the matching variant of integration-v2-error-tracking-step and integration-v2-feature-flags-step. Pages a step shares across every framework, such as the three feature flag guides, are that step's shared_docs and ship in the step skill (#529), not in a row.

A run picks a step skill by framework and takes that family's default. So a test compares each framework's default row with the step's default variant, and every other row with the variant of its own id.

The default Nuxt row

Default for nuxt Error tracking page
The packs nuxt-3-6
integration-v2-error-tracking-step nuxt-4 nuxt-3-7.md

A Nuxt run takes the nuxt-3-6 pack and reads the 3.7 error tracking page today. The nuxt-3-6 row lists that page, so a run reads the same page from its pack. A Nuxt 4 project gets the Nuxt 4 pack once frameworkVariant is a match key in the wizard.

Pages that share a file name

A page was written under the last part of its URL, so error-tracking/installation/python.md and libraries/python.md both became python.md. The skill listed both titles, and the second page replaced the first.

Before Now
Two pages of one folder that share a last part One file, holding the second page Each takes its parent part too: installation-python.md and libraries-python.md
Two pages that would still share a name The second replaces the first The build fails
Every other page next-js.md The same

One writer handles every page, in references/ and in a product's folder. The fix restores a page in:

  • The error tracking step, integration-v2-error-tracking-step, for Expo, FastAPI, SvelteKit, Swift and WordPress, and the packs that carry the same pages.
  • error-tracking-dotnet and error-tracking-ruby-on-rails.
  • feature-flags-dotnet and feature-flags-aspnetcore-feature-management.
  • omnibus-instrument-error-tracking and omnibus-instrument-feature-flags.
  • The marketplace copies of those six skills, in posthog-all, posthog-error-tracking and posthog-feature-flags.

What a pack gains

Addition Where Read by
Each product's pages, one file per page references/<product>/ PR 5, when it completes a step skill
The row as built: its match keys, its rules file, each product's pages with their titles variant.json at the pack's root PR 5
A frameworkVariant:<value> tag The pack's menu entry The wizard's skill resolver, once frameworkVariant is a match key

A product's pages have a folder of their own because two products can have a page of the same name: the React library page and the React error tracking page are both react.md.

SKILL.md lists none of the new files, so it reads as it does today. Every reader of a pack gets the new files with it: 5 to 8 more files a pack, 39 KB unpacked at the median.

Variant values

Fourteen rows name a frameworkVariant. The values are the ids the wizard's framework configs report, and the wizard's trait takes the same name in PostHog/wizard#1559.

Framework Rows and their frameworkVariant
nextjs app-router, pages-router
react-router v6, v7-framework, v7-data, v7-declarative
tanstack-router file-based, code-based
astro static, ssr, hybrid, view-transitions
react-native expo. The react-native row names none
nuxt 4.x on nuxt-4. The nuxt-3-6 row names none

A row with no frameworkVariant serves every value its framework's other rows do not name. The wizard supplies only framework as a match key today, so the new tags change no pick.

What does not change

  • SKILL.md in all 40 packs, byte for byte.
  • Every bundle and skill but the eleven named above, the manifest and the agent prompts.
  • The marketplace has the same plugin skills. The packs it copies carry the new files.
  • The largest pack is 84 KB, and skills-mcp-resources.zip is 13.2 MB against a 16 MB limit.

check:bundle-diff gains --additive, for a release that only adds. It allows a file, a zip entry, a menu entry or a field the baseline lacks, and fails on anything the baseline has that is changed or gone. A list that is empty in the baseline has nothing to keep, so its first records count as additions.

Kept as it is

  • react-vite, javascript_node and javascript_web name their example under example_path, a key the build does not read, so they ship without EXAMPLE.md. Reading it would change their SKILL.md.

Test plan

  • pnpm test: 390 tests pass, 40 of them new
  • pnpm build, then pnpm check:bundle-diff --additive against a build of main: 54 differences, every one in the eleven skills with the page fix and their marketplace copies
  • variant-table.test.js: the build writes the pages and variant.json, leaves SKILL.md and every other file as they were, rejects a row with neither pages nor a gap, gives two pages that share a file name their parent part, rejects two that would still share one, rejects a product listed twice and a match value with a colon, and each row's pages equal the pages of the step variant a run on it reads
  • additive-diff.test.js: what counts as kept, changed, removed and added

🤖 Generated with Claude Code

…er-product pages

Each integration variant lists its error tracking and feature flag pages
under `docs`, taken from the two step skills that ship them today, and
names the variant it is written for under `match`.

The build writes each product's pages to `references/<product>/` in the
pack, lists them in a `variant.json` at the pack's root, and publishes
`match` as `key:value` menu tags. SKILL.md lists none of the new files,
so every pack's SKILL.md is unchanged. The build fails when a row has
neither pages nor a declared gap for a product.

`check:bundle-diff --additive` passes when a build only adds: files, zip
entries, menu entries and fields.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/go-sdk
  • /wizard-ci mcp-analytics/python-sdk
  • /wizard-ci mcp-analytics/ruby-sdk
  • /wizard-ci mcp-analytics/rust-sdk
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a wizard branch:

  • /wizard-ci all wizard:my-branch

Add wizard:<branch> to any command above to pin the wizard branch. It defaults to main.

Results will be posted here when complete.

A `match` key is a trait name as the wizard spells it, and the wizard's
trait for a framework's variant is `frameworkVariant`. The 14 rows that
name a variant use that key, so their packs publish
`frameworkVariant:<value>` tags, and the build rejects a key that is
not a trait name.

Co-Authored-By: Claude <noreply@anthropic.com>
@edwinyjlim
edwinyjlim marked this pull request as ready for review November 13, 2026 22:05
@edwinyjlim
edwinyjlim requested a review from a team as a code owner November 13, 2026 22:05
Comment thread context/skills/integration/config.yaml Resolved

sarahxsanders commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

my agent caught an existing error-tracking docs issue that carries over into the new packs: some error-tracking guides and general sdk guides have the same filename, so the sdk guide overwrites the error-tracking guide during the build.

it confirmed this locally for fastapi (python.md), wordpress (php.md), sveltekit (svelte.md), swift/ios (ios.md), and expo (react-native.md). the pack still lists the error-tracking guide, but the file actually contains the general sdk docs.

this happens in the old packages too, so the before/after checks pass even though those guides are missing. these need distinct filenames, with the references updated to match, so both pages make it into the pack.

@johncwaters johncwaters left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES

See inline comments.

Automated review by Glimmervoid. Not written by a human

Comment thread scripts/lib/tests/variant-table.test.js Resolved
Comment thread scripts/lib/skill-generator.js Resolved

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

Comment thread scripts/check-bundle-diff.js Resolved
Comment thread scripts/lib/skill-generator.js Resolved
edwinyjlim and others added 3 commits November 13, 2026 16:36
…day's Nuxt page

Two pages of a skill were written under the last part of their URL, so
`error-tracking/installation/python.md` and `libraries/python.md` both
became `python.md` and the second replaced the first. The skill listed
both titles and shipped one page. When two pages of a folder share that
part, each now takes its parent part too: `installation-python.md` and
`libraries-python.md`. The build fails when two pages would still be
written under one name. One writer handles every page, in `references/`
and in a product's folder.

This restores a page in the error tracking step for Expo, FastAPI,
SvelteKit, Swift and WordPress, in the packs that carry the same pages,
and in six other skills: error-tracking-dotnet,
error-tracking-ruby-on-rails, feature-flags-dotnet,
feature-flags-aspnetcore-feature-management,
omnibus-instrument-error-tracking and omnibus-instrument-feature-flags.

The default Nuxt row lists the 3.7 error tracking page. A run picks a
step skill by framework and takes that family's default, which for
error tracking is nuxt-4, so that is the page a Nuxt run reads today.
The row test now compares each row with the step variant a run on it
reads.

Co-Authored-By: Claude <noreply@anthropic.com>
`check-bundle-diff --additive` compared a list that is empty in the
baseline as one value, so its first record read as a change. An empty
list has nothing to keep, so everything the candidate lists is an
addition. The comparison moves to scripts/lib/additive-diff.js, with
tests.

Co-Authored-By: Claude <noreply@anthropic.com>
…nd match

A product listed twice, a product name that is not kebab-case, a match
value with a colon and a match that is a list are each rejected, and a
test now says so.

Co-Authored-By: Claude <noreply@anthropic.com>

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

@edwinyjlim
edwinyjlim merged commit 1c25a25 into main Nov 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants