Repository navigation
Restructure and update the autoharness reference page - #4991
Conversation
Move the existing content under the new section layout without changing its wording, and add placeholder comments for the content that is still missing. Also fix the link to the mined invariants section, whose anchor did not match the generated heading id. Towards model-checking#4979
|
@acearyanarun @wodex1nhaoIeng the skeleton is up: the current page moved into Felipe's structure, no wording changes yet. Could you take a look and say whether the layout works for you? If it does, there are three parts to fill in so we don't edit the same lines:
|
Describe what a successful automatic harness does not cover: single monomorphization, bounded arguments, and the opt-in constructor and mined-invariant modes. Add an "Other limitations" subsection for the under-approximations that apply by default without a marker in the summary table, including the closure side effects reported in model-checking#4994. Towards model-checking#4979
Fill in the "Selecting functions" section of the autoharness reference page (towards model-checking#4979): include/exclude pattern matching against instantiated names, --list/--format, the full list of skip reasons, and generic instantiation, including Fn-bound parameters and the SIMD and const-generic restrictions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UFzbxD33o5GHieAP36Da8B
Describe the default harness timeout and unwinding bound, when Kani reminds the user of them, and how to read the selected/skipped tables, the summary table, and its "(bounded)" and "(ctor)" markers. Towards model-checking#4979
Describe the --std option, its path requirement, and that the alloc-dependent argument models are unavailable when verifying the standard library. Also note that concrete playback is not supported. Towards model-checking#4979
Show how the driver and the compiler split the work, from option handling through function selection, harness generation and code generation to running CBMC and printing the summary. Towards model-checking#4979
Update the overview and usage text that still said every argument must implement Arbitrary, align the variadic skip reason with the code, spell "autoharness" consistently, and link the Fn-bound closure models to the closure soundness caveat (model-checking#4994). Towards model-checking#4979
Fill in the bound options, contract preconditions, layout niches, UB and panic checks, contract postconditions and --check-invariants, and smooth out the mined-invariants text split by the restructure. Towards model-checking#4979
Reword the --std model availability note, and make the nested-slice and constructor-generics limits more precise. Towards model-checking#4979
Under --constructor-args, values of a type with its own Arbitrary implementation are still filtered by mined invariants, but the harness is not marked "(ctor)" (model-checking#4995). Say so in the constructor and mined invariant sections, the summary table, and the soundness caveats. Towards model-checking#4979
|
did you review these changes? do they seem helpful? @acearyanarun @Tianshu-Huang @wodex1nhaoIeng @srivatsansamraj |
There was a problem hiding this comment.
🟢 Approval recommended
Only a minor, non-blocking section-link correction remains in this documentation-only change.
1 open issue
What changed in this PR
Updates the autoharness reference to reflect current implementation and organize guidance around selecting functions, generating inputs, and interpreting verification results.
Changes:
- Documents supported argument models, generic instantiation, skip reasons, and CLI options.
- Explains verification defaults, result markers, and soundness limitations.
- Adds a contributor-oriented overview of the driver/compiler pipeline.
| File | Description |
|---|---|
docs/src/reference/experimental/autoharness.md |
Reorganizes and expands the autoharness reference, including modeling limits and architecture. |
Review effort: Balanced
Tianshu-Huang
left a comment
There was a problem hiding this comment.
I checked the page against main and everything matches, and all the #4979 items are in. The anchors are fine too, Yanjun is right about the en dash. Approving.
|
Looks good to me, but since I contributed some commits to this PR, I’m not sure whether it would be appropriate for me to review it. |
|
Hi Felipe, yes, I reviewed the updated documentation, particularly the function-selection and skip-reason sections I contributed to. I noticed two minor improvements: some CLI flags in headings render as en dashes, and the function-selection order could be clarified to include the comptime and no-body checks. Neither appears blocking. Overall, the changes look good to me! |
feliperodri
left a comment
There was a problem hiding this comment.
Thanks for the restructure. The headings follow the structure proposed in #4979, and the claims I spot-checked against the code are accurate (generic instantiation candidates and limits, skip reasons, default timeout/unwind, --list and --std behavior, unbounded slice/Vec models, mined-invariant limits, Fn model shapes).
Requesting changes for the following, in priority order:
- Add the visual overview (inline comment on the "Selecting functions" heading). This is the main change I'd like to see.
- Fix the imprecise selection-order sentence (inline comment).
- Make the two fragile heading anchors robust (inline comments).
- Confirm the docs build. Only the "Auto Label" check has reported on this PR, so no CI job has built the book. Please run
mdbook build docslocally and confirm there are no new warnings and no broken links, or get the docs workflow approved to run. - Optional, not blocking: the page is now ~790 lines. Consider a follow-up that splits it into overview, argument generation, and soundness pages.
I'll re-review once these are addressed.
Describe the selection checks in the order automatic_harness_partition runs them: Kani-internal and compile-time-only functions are rejected before generic instantiation, followed by the body, proof-harness, include/exclude, variadic and argument checks. State that the skip-reason table follows this order. Rename the bounded-arguments, constructor-args and --list headings so their ids no longer depend on how mdBook treats "--" and other punctuation, and update all links. This also removes the near-duplicate with the "Bounded arguments" soundness caveat.
Add the three-phase SELECT / GENERATE / VERIFY & REPORT diagram from model-checking#4979 before "Selecting functions", with links to the section for each phase, the soundness caveats and the architecture section. The diagram marks mined invariants as applying only under --constructor-args, as the page describes.
… docs/autoharness-restructure
Render the diagram as a raw HTML <pre> block so the box titles can link to their sections, as asked in review, and drop the separate link list. Restore the diagram text to the model-checking#4979 original apart from the mined invariants line.
|
Thanks for the careful review! All items are addressed:
|
…rguments (model-checking#5064) This fixes model-checking#4811, an issue in the Autoharness milestone. **Priority: Low (L3, a coverage gap under 100 functions).** This adds automatic harnesses, under `--bounded-arguments`, for 27 standard-library functions that take `CharArraySearcher`, `CharArrayRefSearcher` or `CharSliceSearcher` arguments; nothing fails while it waits. With `--bounded-arguments`, autoharness can now generate `CharArraySearcher<'_, N>`, `CharArrayRefSearcher<'_, '_, N>` and `CharSliceSearcher<'_, '_>` arguments, by value, `&` or `&mut`. These are the searchers for `[char; N]`, `&[char; N]` and `&[char]` patterns in `core::str::pattern`. ### Changes 1. New `kani_core` models build the searcher through the public `Pattern::into_searcher` API over a bounded nondeterministic haystack (from `any_str_ref`, string bound) and a nondeterministic needle (`[char; N]`, or a `&[char]` of up to the slice bound for `CharSliceSearcher`). They then advance it a nondeterministic number of steps from each end, so that searchers in partially advanced states are also generated. 2. `multi_char_eq_searcher_kind` identifies the three types by definition path and is shared by the eligibility check and harness generation. 3. The searchers' `Debug::fmt` methods are verified as ordinary functions rather than through the formatting-trait model, since the searcher needs harness-owned storage. 4. The storage-array construction of the slice/str branches is factored into `build_any_array_storage_ref`. The needle storage of `CharArrayRefSearcher` and `CharSliceSearcher` is generated with a single `kani::any()` call (`build_any_whole_array_storage_ref`), so a large `N` does not grow the harness body. 5. The reference page and the `--slice-bound`/`--string-bound` help text document the new argument types. ### Testing 1. New test `cargo_autoharness_multichareq_searcher`: 7 harnesses, 6 pass and 1 fails as expected, and all 33 cover properties are satisfied. It takes about 4 minutes locally with `--string-bound 4 --slice-bound 4`; the slowest harness (`char_slice_value`) takes about 70s. The script uses `--harness-timeout 10m` rather than the usual 5m because that harness came close to the 5m limit on `macos-15-intel`. Timing is driven by the string bound (haystack size): lowering the slice bound from 4 to 2 made no measurable difference, and a string bound of 3 would drop 4-byte UTF-8 characters from the haystack. 2. All 42 autoharness regression tests pass. `cargo fmt --check` and `cargo clippy --workspace --tests -- -D warnings` pass. ### Results on verify-std 1. With `--bounded-arguments`, all 27 functions listed in model-checking#4811 now get harnesses (9 per searcher type). The other 23 functions in `pattern.rs` (`CharPredicateSearcher`, `CharSearcher`, `StrSearcher`) are still skipped and are out of scope here. 2. Verifying the 27 harnesses (release build, string bound 4, slice bound 16, 10-minute timeout each): 8 fail and 19 time out. 3. The 8 failures are the preconditions of `char::from_u32_unchecked` and `hint::unreachable_unchecked`, caused by model-checking#5030 (`&str` from `any_str_ref` can be invalid UTF-8 under verify-std). CBMC traces for `CharArraySearcher::haystack` and `CharArraySearcher::next` confirm this: the haystack bytes are mis-validated by `from_utf8`, and the failures occur while the model advances the searcher, before the function under test runs. For `next`, the call graph shows the `unreachable_unchecked` check is reachable only from UTF-8 decoding. The other 6 fail on the same two checks and very likely share the cause, but I did not trace them individually. ### Notes 1. This PR and model-checking#4991 both edit `docs/src/reference/experimental/autoharness.md`. I'll resolve the conflict in whichever merges second. 2. Related: model-checking/verify-rust-std#722 adds hand-written harnesses for these searchers and builds haystacks with `encode_utf8`. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Resolves #4979.
This rewrites the autoharness reference page (
docs/src/reference/experimental/autoharness.md) to match the implementation onmain, following the structure proposed in #4979: selecting functions, generating harnesses (arguments, assumptions, checks), verifying and reading results, soundness caveats, and an architecture overview for contributors.How the items in #4979 are addressed:
Vec: new "Primitive slices and Vec" section; "Bounded Arguments" and "Arguments Implementing Arbitrary" now distinguish the primitive unbounded models from the bounded generators.Fn-bound generics: "Generic Functions" (instantiation) and "Fn-bound closures" (the model and its limits).Box/Rc/Arc: new "Box, Rc and Arc" section.--stdin "Usage",--list --formatin "Listing functions (--list)", and the bound flags in "Bounded Arguments".The soundness caveats are now collected in one section: single monomorphization, aliasing between arguments, bounded arguments, vacuous constructors, and the mined-invariant heuristic. That section also has an "Other limitations" subsection that is not in the proposed structure. It lists the under-approximations that apply by default without a marker in the summary table. Happy to drop or move it if you prefer.
While writing the page we found and reported two issues, both described in the caveats:
Fn-bound closure models have no side effects, so callbacks that modify shared state are not covered.--constructor-args, mined-invariant filtering of a type with its ownArbitraryimplementation is not marked "(ctor)".Other fixes:
Arbitrary.Notes for review:
--alias-arguments.Contributors:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.