Skip to content

Restructure and update the autoharness reference page - #4991

Merged
feliperodri merged 20 commits into
model-checking:mainfrom
CYJ904:docs/autoharness-restructure
Nov 3, 2026
Merged

feliperodri merged 20 commits into
model-checking:mainfrom
CYJ904:docs/autoharness-restructure

Conversation

@CYJ904

@CYJ904 CYJ904 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #4979.

This rewrites the autoharness reference page (docs/src/reference/experimental/autoharness.md) to match the implementation on main, following the structure proposed in #4979: selecting functions, generating harnesses (arguments, assumptions, checks), verifying and reading results, soundness caveats, and an architecture overview for contributors.

How the items in #4979 are addressed:

  1. Unbounded slices and Vec: new "Primitive slices and Vec" section; "Bounded Arguments" and "Arguments Implementing Arbitrary" now distinguish the primitive unbounded models from the bounded generators.
  2. Fn-bound generics: "Generic Functions" (instantiation) and "Fn-bound closures" (the model and its limits).
  3. Box/Rc/Arc: new "Box, Rc and Arc" section.
  4. Layout niches: new "Layout niches" section under Assumptions.
  5. Default settings: new "Default settings" section (60s timeout, unwind 20, and the reminder printed when a harness fails).
  6. Skip reasons: new "Skip reasons" section listing every reason shown to users, plus the hidden "Kani implementation" case.
  7. SIMD and const-generic rejections: "Generic Functions".
  8. CLI items: --std in "Usage", --list --format in "Listing functions (--list)", and the bound flags in "Bounded Arguments".

The soundness caveats are now collected in one section: single monomorphization, aliasing between arguments, bounded arguments, vacuous constructors, and the mined-invariant heuristic. That section also has an "Other limitations" subsection that is not in the proposed structure. It lists the under-approximations that apply by default without a marker in the summary table. Happy to drop or move it if you prefer.

While writing the page we found and reported two issues, both described in the caveats:

  1. Autoharness: Fn-bound closure models have no side effects, so callbacks that modify shared state are not covered #4994: the Fn-bound closure models have no side effects, so callbacks that modify shared state are not covered.
  2. Autoharness --constructor-args: mined-invariant filtering of types with their own Arbitrary is not marked "(ctor)" #4995: under --constructor-args, mined-invariant filtering of a type with its own Arbitrary implementation is not marked "(ctor)".

Other fixes:

  1. A link to the mined invariants section whose anchor did not match the generated heading id.
  2. The overview and usage text that still said every argument must implement Arbitrary.

Notes for review:

  1. If Autoharness: fail a harness that no generated input reaches #4978 merges first, the "Vacuous constructors" caveat should be updated or removed. If Autoharness: add --alias-arguments to model aliasing between arguments #4958 merges first, the aliasing caveat should point to --alias-arguments.
  2. The page builds with mdBook without new warnings, and all same-page links resolve.

Contributors:

  1. Selecting functions: @acearyanarun
  2. Generating harnesses: @wodex1nhaoIeng
  3. Skeleton, usage, verifying and reading results, soundness caveats, architecture overview, and integration: @CYJ904

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Move the existing content under the new section layout without
changing its wording, and add placeholder comments for the content
that is still missing. Also fix the link to the mined invariants
section, whose anchor did not match the generated heading id.

Towards model-checking#4979
@CYJ904

CYJ904 commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

@acearyanarun @wodex1nhaoIeng the skeleton is up: the current page moved into Felipe's structure, no wording changes yet. Could you take a look and say whether the layout works for you?

If it does, there are three parts to fill in so we don't edit the same lines:

  1. Selecting functions (skip reasons, generic instantiation, --list/--format)
  2. Generating harnesses (arguments, assumptions, checks)
  3. Usage, results, soundness caveats, architecture

Describe what a successful automatic harness does not cover: single
monomorphization, bounded arguments, and the opt-in constructor and
mined-invariant modes. Add an "Other limitations" subsection for the
under-approximations that apply by default without a marker in the
summary table, including the closure side effects reported in model-checking#4994.

Towards model-checking#4979
CYJ904 and others added 8 commits October 8, 2026 17:46
Fill in the "Selecting functions" section of the autoharness reference page
(towards model-checking#4979): include/exclude pattern matching against instantiated names,
--list/--format, the full list of skip reasons, and generic instantiation,
including Fn-bound parameters and the SIMD and const-generic restrictions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFzbxD33o5GHieAP36Da8B
Describe the default harness timeout and unwinding bound, when Kani
reminds the user of them, and how to read the selected/skipped tables,
the summary table, and its "(bounded)" and "(ctor)" markers.

Towards model-checking#4979
Describe the --std option, its path requirement, and that the
alloc-dependent argument models are unavailable when verifying the
standard library. Also note that concrete playback is not supported.

Towards model-checking#4979
Show how the driver and the compiler split the work, from option
handling through function selection, harness generation and code
generation to running CBMC and printing the summary.

Towards model-checking#4979
Update the overview and usage text that still said every argument must
implement Arbitrary, align the variadic skip reason with the code,
spell "autoharness" consistently, and link the Fn-bound closure models
to the closure soundness caveat (model-checking#4994).

Towards model-checking#4979
Fill in the bound options, contract preconditions, layout niches,
UB and panic checks, contract postconditions and --check-invariants,
and smooth out the mined-invariants text split by the restructure.

Towards model-checking#4979
Reword the --std model availability note, and make the nested-slice and
constructor-generics limits more precise.

Towards model-checking#4979
Under --constructor-args, values of a type with its own Arbitrary
implementation are still filtered by mined invariants, but the harness
is not marked "(ctor)" (model-checking#4995). Say so in the constructor and mined
invariant sections, the summary table, and the soundness caveats.

Towards model-checking#4979
@CYJ904 CYJ904 changed the title Restructure the autoharness reference page Restructure and update the autoharness reference page Oct 9, 2026
@CYJ904
CYJ904 marked this pull request as ready for review October 9, 2026 05:53
@CYJ904
CYJ904 requested review from a team as code owners October 9, 2026 05:53
@feliperodri

Copy link
Copy Markdown
Member

did you review these changes? do they seem helpful? @acearyanarun @Tianshu-Huang @wodex1nhaoIeng @srivatsansamraj

@feliperodri
feliperodri requested a balanced review from Copilot October 22, 2026 03:02
@feliperodri feliperodri self-assigned this Oct 22, 2026
@feliperodri feliperodri added the [C] Documentation Additions and improvements to our documentation label Oct 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Only a minor, non-blocking section-link correction remains in this documentation-only change.

1 open issue
What changed in this PR

Updates the autoharness reference to reflect current implementation and organize guidance around selecting functions, generating inputs, and interpreting verification results.

Changes:

  • Documents supported argument models, generic instantiation, skip reasons, and CLI options.
  • Explains verification defaults, result markers, and soundness limitations.
  • Adds a contributor-oriented overview of the driver/compiler pipeline.
File Description
docs/​src/​reference/​experimental/​autoharness.md Reorganizes and expands the autoharness reference, including modeling limits and architecture.

Review effort: Balanced

@Tianshu-Huang Tianshu-Huang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the page against main and everything matches, and all the #4979 items are in. The anchors are fine too, Yanjun is right about the en dash. Approving.

@wodex1nhaoIeng

Copy link
Copy Markdown
Contributor

Looks good to me, but since I contributed some commits to this PR, I’m not sure whether it would be appropriate for me to review it.

@acearyanarun

Copy link
Copy Markdown
Contributor

Hi Felipe, yes, I reviewed the updated documentation, particularly the function-selection and skip-reason sections I contributed to.

I noticed two minor improvements: some CLI flags in headings render as en dashes, and the function-selection order could be clarified to include the comptime and no-body checks. Neither appears blocking.

Overall, the changes look good to me!

@feliperodri feliperodri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the restructure. The headings follow the structure proposed in #4979, and the claims I spot-checked against the code are accurate (generic instantiation candidates and limits, skip reasons, default timeout/unwind, --list and --std behavior, unbounded slice/Vec models, mined-invariant limits, Fn model shapes).

Requesting changes for the following, in priority order:

  1. Add the visual overview (inline comment on the "Selecting functions" heading). This is the main change I'd like to see.
  2. Fix the imprecise selection-order sentence (inline comment).
  3. Make the two fragile heading anchors robust (inline comments).
  4. Confirm the docs build. Only the "Auto Label" check has reported on this PR, so no CI job has built the book. Please run mdbook build docs locally and confirm there are no new warnings and no broken links, or get the docs workflow approved to run.
  5. Optional, not blocking: the page is now ~790 lines. Consider a follow-up that splits it into overview, argument generation, and soundness pages.

I'll re-review once these are addressed.

Describe the selection checks in the order automatic_harness_partition
runs them: Kani-internal and compile-time-only functions are rejected
before generic instantiation, followed by the body, proof-harness,
include/exclude, variadic and argument checks. State that the
skip-reason table follows this order.

Rename the bounded-arguments, constructor-args and --list headings so
their ids no longer depend on how mdBook treats "--" and other
punctuation, and update all links. This also removes the
near-duplicate with the "Bounded arguments" soundness caveat.
Add the three-phase SELECT / GENERATE / VERIFY & REPORT diagram from
model-checking#4979 before "Selecting functions", with links to the section for each
phase, the soundness caveats and the architecture section. The diagram
marks mined invariants as applying only under --constructor-args, as
the page describes.
Render the diagram as a raw HTML <pre> block so the box titles can link
to their sections, as asked in review, and drop the separate link list.
Restore the diagram text to the model-checking#4979 original apart from the mined
invariants line.
@CYJ904

CYJ904 commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review! All items are addressed:

  1. Visual overview: new "How it works" section with the Autoharness documentation is out of date and missing several features #4979 diagram; each box links to its section.
  2. Selection order: the paragraph now follows automatic_harness_partition exactly.
  3. Heading anchors: both headings (and "Listing functions") no longer have punctuation that depends on mdBook settings, and all links are updated.
  4. Docs build: built locally with mdbook 0.5.2 and mdbook-graphviz 0.3.1 on the final version and compared against main. No new warnings (the 4 existing ones in attributes.md and faq.md are on main too) and no new broken links; a link check over the generated HTML finds 0 broken links on the autoharness page. The PR also fixes one broken link that main has (#mined-invariants---check-invariants).
  5. Happy to do the split in a follow-up PR once this one lands.

@CYJ904
CYJ904 requested a review from feliperodri November 1, 2026 06:55
@feliperodri
feliperodri enabled auto-merge November 2, 2026 22:56
@feliperodri feliperodri removed their assignment Nov 2, 2026
@feliperodri
feliperodri added this pull request to the merge queue Nov 2, 2026
Merged via the queue into model-checking:main with commit 044b332 Nov 3, 2026
33 checks passed
wodex1nhaoIeng pushed a commit to wodex1nhaoIeng/kani that referenced this pull request Nov 10, 2026
…rguments (model-checking#5064)

This fixes model-checking#4811, an issue in the Autoharness milestone.

**Priority: Low (L3, a coverage gap under 100 functions).** This adds
automatic harnesses, under `--bounded-arguments`, for 27
standard-library functions that take `CharArraySearcher`,
`CharArrayRefSearcher` or `CharSliceSearcher` arguments; nothing fails
while it waits.

With `--bounded-arguments`, autoharness can now generate
`CharArraySearcher<'_, N>`, `CharArrayRefSearcher<'_, '_, N>` and
`CharSliceSearcher<'_, '_>` arguments, by value, `&` or `&mut`. These
are the searchers for `[char; N]`, `&[char; N]` and `&[char]` patterns
in `core::str::pattern`.

### Changes

1. New `kani_core` models build the searcher through the public
`Pattern::into_searcher` API over a bounded nondeterministic haystack
(from `any_str_ref`, string bound) and a nondeterministic needle
(`[char; N]`, or a `&[char]` of up to the slice bound for
`CharSliceSearcher`). They then advance it a nondeterministic number of
steps from each end, so that searchers in partially advanced states are
also generated.
2. `multi_char_eq_searcher_kind` identifies the three types by
definition path and is shared by the eligibility check and harness
generation.
3. The searchers' `Debug::fmt` methods are verified as ordinary
functions rather than through the formatting-trait model, since the
searcher needs harness-owned storage.
4. The storage-array construction of the slice/str branches is factored
into `build_any_array_storage_ref`. The needle storage of
`CharArrayRefSearcher` and `CharSliceSearcher` is generated with a
single `kani::any()` call (`build_any_whole_array_storage_ref`), so a
large `N` does not grow the harness body.
5. The reference page and the `--slice-bound`/`--string-bound` help text
document the new argument types.

### Testing

1. New test `cargo_autoharness_multichareq_searcher`: 7 harnesses, 6
pass and 1 fails as expected, and all 33 cover properties are satisfied.
It takes about 4 minutes locally with `--string-bound 4 --slice-bound
4`; the slowest harness (`char_slice_value`) takes about 70s. The script
uses `--harness-timeout 10m` rather than the usual 5m because that
harness came close to the 5m limit on `macos-15-intel`. Timing is driven
by the string bound (haystack size): lowering the slice bound from 4 to
2 made no measurable difference, and a string bound of 3 would drop
4-byte UTF-8 characters from the haystack.
2. All 42 autoharness regression tests pass. `cargo fmt --check` and
`cargo clippy --workspace --tests -- -D warnings` pass.

### Results on verify-std

1. With `--bounded-arguments`, all 27 functions listed in model-checking#4811 now get
harnesses (9 per searcher type). The other 23 functions in `pattern.rs`
(`CharPredicateSearcher`, `CharSearcher`, `StrSearcher`) are still
skipped and are out of scope here.
2. Verifying the 27 harnesses (release build, string bound 4, slice
bound 16, 10-minute timeout each): 8 fail and 19 time out.
3. The 8 failures are the preconditions of `char::from_u32_unchecked`
and `hint::unreachable_unchecked`, caused by model-checking#5030 (`&str` from
`any_str_ref` can be invalid UTF-8 under verify-std). CBMC traces for
`CharArraySearcher::haystack` and `CharArraySearcher::next` confirm
this: the haystack bytes are mis-validated by `from_utf8`, and the
failures occur while the model advances the searcher, before the
function under test runs. For `next`, the call graph shows the
`unreachable_unchecked` check is reachable only from UTF-8 decoding. The
other 6 fail on the same two checks and very likely share the cause, but
I did not trace them individually.

### Notes

1. This PR and model-checking#4991 both edit
`docs/src/reference/experimental/autoharness.md`. I'll resolve the
conflict in whichever merges second.
2. Related: model-checking/verify-rust-std#722 adds hand-written
harnesses for these searchers and builds haystacks with `encode_utf8`.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[C] Documentation Additions and improvements to our documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Autoharness documentation is out of date and missing several features

6 participants