docs(skills): verify a fix at the layer its symptom lives in - #70
Merged
Conversation
Adds .claude/skills/verify-in-runtime.md and the checklist rule that routes to it, so browser-level verification is a normal step for the bugs that need it rather than something rediscovered per incident. The motivating pattern: twice this week a green test suite coexisted with a live bug. In mendixlabs#812 a duplicate codec.RegisterTypeDefaults for the same $Type silently clobbered the fix — registrations overwrite rather than merge and resolve by init order — so the unit test passed while the emitted BSON stayed wrong. In mendixlabs#808 the integration test had only ever skipped, for want of an mx binary, and a skip reads exactly like a pass. A green suite is evidence about the layer it tests and nothing more. mendixlabs#812 is the case the new tier exists for: every popup opened by an mxcli-authored button rendered a blank caption, yet the BSON was structurally valid, mx check reported 0 errors and MxBuild completed. Nothing below the browser could see it, because the defect WAS the rendering — an empty Microflows$TextTemplate is not "no title override", it is an override to the empty string. The trigger rule is deliberately narrow: use the layer the symptom lives in. Parser to unit test, BSON to a test on the encoded document, on-disk effects to an integration test, and only render-time behaviour to the runtime tier. Of four fixes this week exactly one qualified; mendixlabs#808, mendixlabs#779 and the semicolon change did not. The skill records the procedure and, more usefully, the traps that cost the most time: cache MxBuild for the version under test explicitly; create the scratch project at a SHORT path, because Mendix throws PathTooLongException on deep ones and in Go tests that turns into a silent skip (set TMPDIR); install the Playwright package but never the browsers; and A/B against a pre-fix binary, since a run that only shows the fixed state demonstrates the app works, not that the change is why. Also the mundane one that nearly landed in a commit: restore files with absolute paths after a mutation test, and finish with git status. Adds a companion checklist item requiring every bug fix to demonstrate its test detects the bug, by reverting the fix and watching it fail. Contributor-only: the top-level .claude/skills/ are not synced into user projects by mxcli init (only .claude/skills/mendix/ is). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
Every bug fix touches .claude/skills/fix-issue.md, and new rows went in directly under the table header. Two branches fixing unrelated bugs therefore inserted at the same line, which git cannot merge — a conflict by construction rather than by bad luck. It cost five separate resolution rounds in one week (#59-#62 three times, then #68 and #69), and each round risks silently dropping a row: one earlier round was "resolved" by deleting a PR's rows entirely and restoring them in a follow-up. Appending puts each branch's insert at a different offset, which git merges without help. The table is unordered — it is looked up by matching a symptom, not read top to bottom — so position carries no meaning and appending costs nothing. Recorded in both places a contributor might look: the skill's How to Use, with the reasoning so it is not "fixed" back, and the PR checklist item in CLAUDE.md. Existing rows are deliberately left where they are. Reordering them would conflict with every open branch at once, which is the problem this change exists to avoid. Folded into this branch rather than opened separately: it edits the same How to Use block, so a sixth branch would have manufactured exactly the conflict it is meant to prevent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
ako
pushed a commit
that referenced
this pull request
Aug 1, 2026
#68 landed, putting its symptom row at the top of the table — the same line this branch inserts at, so the conflict recurred immediately after the previous resolution. That is the pattern, not bad luck. Resolved by keeping main's rows and moving this branch's row to the END of the table, which is the convention #70 introduces. Applying it here rather than waiting: this branch is the exact case it exists for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. Adds
.claude/skills/verify-in-runtime.mdand the checklist rule that routes to it, so browser-level verification is a normal step for the bugs that need it rather than something rediscovered per incident.Why
Twice this week a green test suite coexisted with a live bug:
codec.RegisterTypeDefaultsfor the same$Typesilently clobbered the fix (registrations overwrite rather than merge, resolved by init order). The unit test passed; the emitted BSON was still wrong.mxbinary. A skip reads exactly like a pass in CI output.A green suite is evidence about the layer it tests, and nothing more.
mendixlabs#812 is the case the new tier exists for: every popup opened by an mxcli-authored button rendered a blank caption, yet the BSON was structurally valid,
mx checkreported 0 errors, and MxBuild completed. Nothing below the browser could see it — the defect was the rendering. An emptyMicroflows$TextTemplateis not "no title override", it is an override to the empty string.The trigger rule — deliberately narrow
mxruns-tags integration)Of four fixes this week exactly one qualified. mendixlabs#808 (visible on disk), mendixlabs#779 (visible in BSON round-trip) and the mandatory-semicolon change (parser) did not. This is not a blanket mandate — the runtime tier costs ~5 min and 1.4 GB per Mendix version.
What's in the skill
The procedure, and more usefully the traps that cost the most time:
PathTooLongExceptionon deep ones, and in Go tests that becomes a silent skip (setTMPDIR)/opt/pw-browsers/chromiumis pre-installed)cpafter acdsilently fails and leaves reverted code in the treeThe finished shape, from mendixlabs#812:
TitleOverrideheaderTextnull"×\nOrder Details Popup""Order Details Popup"TextTemplate"×"""Also adds
A companion checklist item requiring every bug fix to demonstrate its test detects the bug — revert the fix, watch it fail. That is what would have caught both failures above.
Contributor-only: top-level
.claude/skills/are not synced into user projects bymxcli init(only.claude/skills/mendix/is). Verified —make sync-skillsdoes not pick it up.CI wiring for a
runtimetest tier is deliberately not included; worth doing once a second bug actually needs it, rather than building the tier speculatively off a sample of one.🤖 Generated with Claude Code
https://claude.ai/code/session_012XR649rKk68z6gBpngu6MA
Generated by Claude Code