Skip to content

fix(public): make docs integration tests consumer-independent - #4016

Merged
PierreBrisorgueil merged 2 commits into
masterfrom
fix/4015-docs-tests-consumer-independent
Aug 4, 2026
Merged

fix(public): make docs integration tests consumer-independent#4016
PierreBrisorgueil merged 2 commits into
masterfrom
fix/4015-docs-tests-consumer-independent

Conversation

@PierreBrisorgueil

@PierreBrisorgueil PierreBrisorgueil commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • What changed: modules/public/tests/public.docs.integration.tests.js — every test that read real on-disk guide content or counts now runs against a controlled, self-authored fixture guide set (config.files.guides fully replaced for the duration of the block, restored in afterAll), or was scoped to a status/shape-only assertion where content is genuinely irrelevant. The raw-markdown :slug.md test now targets a fixture guide at a unique slug and asserts its own content.
  • Why: a consumer shipping its own guide at a globally-shared slug (e.g. welcome/quickstart) deterministically wins that slug by design (an earlier, already-merged change). Any assertion in this suite that was pinned to which real guide wins a shared slug was a consumer-dependence bug by construction — it would fail in a consumer's own test run even though nothing was actually broken. This completes the same fix applied to one describe block in an earlier, already-merged PR, extended here to the rest of the file.
  • Related issues: Closes 🐛 fix(public): docs integration tests depend on real on-disk guides — make the file consumer-independent #4015

Scope

  • Module(s) impacted: public (docs)
  • Cross-module impact: none
  • Risk level: low

Validation

  • npm run lint
  • npm test (integration + unit, public-scoped)
  • Manual checks done (if applicable) — ran the suite once normally, then once with a synthetic extra guide file injected at the welcome/quickstart slugs from a fake non-core module (created under a throwaway module directory, removed immediately after): identical green result both times, confirming the suite no longer depends on which real guide wins a shared slug.

Guardrails check

  • No secrets or credentials introduced (.env*, secrets/**, keys, tokens)
  • No risky rename/move of core stack paths
  • Changes remain merge-friendly for downstream projects
  • Tests added or updated when behavior changed

Notes for reviewers

  • Security considerations: none — test-file-only change, zero source/runtime changes.
  • Mergeability considerations: the existing slug-collision precedence describe block (added by an earlier, already-merged PR) is untouched; this PR only restructures the two describe blocks preceding it in the same file.
  • Follow-up tasks (optional): none.

Summary by CodeRabbit

  • Tests
    • Improved guide integration test reliability by using controlled temporary content instead of repository-provided guides.
    • Separated general response checks from content-specific validations.
    • Added cleanup and cache-reset handling to prevent test state from affecting other tests.
    • Retained coverage for guide navigation, metadata, raw Markdown content, and unknown guide slugs.

The public docs integration suite still read real on-disk guide
content/counts in several places (tree grouping, the raw-markdown
:slug.md test, the front-matter leak check). A consumer shipping its
own guide at a globally-shared slug (welcome/quickstart) now
deterministically wins that slug by design (#4011), so any assertion
pinned to which real guide wins is a consumer-dependence bug.

Split the file's first describe block into a shape/status-only block
(no fixture needed, content is irrelevant) and a controlled-fixture
block that fully replaces config.files.guides with self-authored
guides (the #4014 pattern), restored in afterAll. The raw-markdown
test now targets a fixture guide at a unique slug and asserts its own
content. The slug-collision precedence block is untouched.

Verified the invariant holds by running the suite with a synthetic
extra guide file injected at the welcome/quickstart slugs from a fake
non-core module — identical green result with or without it.

Closes #4015
@PierreBrisorgueil PierreBrisorgueil added Fix A bug fix Tests Adding missing tests or correcting existing labels Aug 4, 2026
@PierreBrisorgueil PierreBrisorgueil self-assigned this Aug 4, 2026
@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@PierreBrisorgueil, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 44 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c4c829a6-44ee-4c54-98f7-6ac009b8a526

📥 Commits

Reviewing files that changed from the base of the PR and between 939821c and 771c9df.

📒 Files selected for processing (1)
  • modules/public/tests/public.docs.integration.tests.js

Walkthrough

The integration tests now separate status and shape checks from content checks. Content checks use temporary guides configured through config.files.guides, with cache clearing and teardown restoration. Assertions no longer depend on shipped guide files or their content.

Changes

Public docs integration tests

Layer / File(s) Summary
Controlled guide fixture isolation
modules/public/tests/public.docs.integration.tests.js
The tests create temporary guides, replace config.files.guides, initialize the application, clear service caches, and restore configuration and files during teardown.
Fixture-based guide assertions
modules/public/tests/public.docs.integration.tests.js
Guide tree, metadata, raw Markdown, and front-matter assertions now use sample-intro and sample-next fixtures. Generic status, shape, and unknown-slug checks remain independent of shipped guide content.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: making public documentation integration tests independent of consumer guides.
Description check ✅ Passed The description covers the required summary, scope, validation, guardrails, related issue, and reviewer notes.
Linked Issues check ✅ Passed The changes address issue #4015 by using controlled fixtures, unique slugs, configuration restoration, and content-independent assertions.
Out of Scope Changes check ✅ Passed The changes are limited to the specified public documentation integration test file and introduce no source or unrelated changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/4015-docs-tests-consumer-independent

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@PierreBrisorgueil
PierreBrisorgueil marked this pull request as ready for review August 4, 2026 18:26
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.54%. Comparing base (05f3621) to head (771c9df).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4016   +/-   ##
=======================================
  Coverage   93.54%   93.54%           
=======================================
  Files         170      170           
  Lines        5759     5759           
  Branches     1846     1846           
=======================================
  Hits         5387     5387           
  Misses        302      302           
  Partials       70       70           
Flag Coverage Δ
integration 61.88% <ø> (+0.05%) ⬆️
unit 76.12% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05f3621...771c9df. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@modules/public/tests/public.docs.integration.tests.js`:
- Around line 8-10: Move the unknown-slug assertion near the test using the
controlled guide fixture, rather than the configuration-independent tests.
Update that test to query a slug explicitly absent from the fixture list,
preserving the expected not-found response without relying on consumer-provided
guide configuration.
- Around line 82-85: Update the fixture written by fs.writeFileSync in the docs
integration test to begin with distinctive YAML front matter, then strengthen
the assertion around the response at the applicable test near lines 172-175 to
verify that marker is absent. Keep the existing documentation-content assertions
intact.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d161165b-8187-4171-9425-09e9a2438f7c

📥 Commits

Reviewing files that changed from the base of the PR and between 05f3621 and 939821c.

📒 Files selected for processing (1)
  • modules/public/tests/public.docs.integration.tests.js

Comment thread modules/public/tests/public.docs.integration.tests.js
Comment thread modules/public/tests/public.docs.integration.tests.js
CodeRabbit review on #4016: the 404-for-unknown-slug test lived in the
shape/status-only block and queried a hardcoded slug against whatever
config.files.guides resolves to at boot. A consumer could in principle
configure a guide at that exact slug, making the test consumer-dependent
again. Move it into the controlled-fixture block (config.files.guides
fully replaced there) and query a slug guaranteed absent from that
fixture set.
@PierreBrisorgueil
PierreBrisorgueil merged commit 585d5c6 into master Aug 4, 2026
8 checks passed
@PierreBrisorgueil
PierreBrisorgueil deleted the fix/4015-docs-tests-consumer-independent branch August 4, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Fix A bug fix Tests Adding missing tests or correcting existing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 fix(public): docs integration tests depend on real on-disk guides — make the file consumer-independent

1 participant