Repository navigation
docs(dsql): add limits/error-code reference, fix wrong messages - #279
Conversation
d43958d to
a136fb4
Compare
a136fb4 to
703c080
Compare
10a0809 to
924d8e8
Compare
924d8e8 to
2706eca
Compare
|
Second review pass against a live cluster.
Also corrected the eval README's prompt and assertion counts and two stale wordings.
|
Code reviewThe corrected error messages, the limits and rejection tables, and the plan-shape guidance hold up on a live cluster. Four behavioral claims did not hold up, and three of them were added or changed in the last two self-review passes: (1) OC001 does hit read-only transactions, (2) OC001 comes from a stale per-session catalog cache, not from a snapshot that predates the change, (3) the ~430 B per index entry overhead and (4) the ~1,955 / ~1,455 key-size figures. The eval-3 grader also passes wrong answers and fails correct ones. Blocking: #1, #2, #4. Every DSQL claim below was re-run against a single-Region PostgreSQL 16 cluster (us-east-1) on 2026-10-06. All test objects used a
Findings dropped during validation (21)
Self-review passes and driftEight force-pushed revisions; the base has not moved (merge-base da51970 =
Drift: the claims that changed most across revisions are the ones that fail on the cluster. OC001 semantics changed four times, the read-only exemption was added at the last head, the 10 MiB model flipped from "unpredictable" to a formula at e62a465, and key size changed four times. Each pass fixed the contradictions the previous one introduced (e62a465 still said "concurrent DDL" in troubleshooting.md), but the passes were checked against themselves, not re-measured. Assessment: structure and direction are sound, and the messages, tables and plan guidance are good. This needs one more targeted pass, not a rewrite. Fix #1–#6 against a cluster, move the PK-inclusion check to the judge, split the limits section out to bring SKILL.md back under 300 lines, and attach a real functional-eval run. Reviewed by 21 sub-agents: code-review 4a–4e, silent-failure-hunter, comment-analyzer, code-simplifier (report-only), pr-test-analyzer, type-design-analyzer, pr-review-toolkit code-reviewer, superpowers code-reviewer, security, drift, PR-body, language-idiomatic, authoring-style, AGENTS.md compliance, prior-comment resolution, commit audit, DSQL docs-claims. Findings were consolidated after all agents returned, then validated against the head SHA, a live DSQL cluster, 🤖 Generated with Claude Code |
OC001 hits reads too, so drop the read-only exemption. The cause is a stale per-session catalog cache, not a transaction snapshot; eval 26 now accepts the documented completed-before-start case. Drop the 10 MiB formula. Per-entry cost depends on index key and PK width, so point agents at the 54000 DETAIL instead. Every index counts toward the 24, UNIQUE included. Key budget is ~1,981 bytes less ~7 per extra column, 1,474 with a 500-byte PK. Add the 2,000-byte DDL-time key check. Zero lint diagnostics is not acceptance; drop the fixed_sql compare. The EOF rule now covers autocommit writes. Fix the capitalisation note and trim SKILL.md to 299 lines. Fix INDEXES_PER_TABLE misgrades in both directions and pin each probe in tests.
The INDEXES_PER_TABLE grader still misgraded close variants of the review probes. "up to 24 hours" after an index noun passed as the cap, and correct answers failed: one naming DSQL rather than the table, the no-primary-key caveat on its own (its "no" read as a negation), and "23 secondary plus the primary key, 24 in total". Each now grades correctly; five new test strings cover them. occ-retry-patterns.md now links to the confirm-or-stop rule in troubleshooting.md instead of offering idempotency as an alternative to confirming. development-guide.md trims the index limit to one clause.
The INDEXES_PER_TABLE regex tried to veto every phrasing that puts
the primary key outside the 24, and each probe round found another
that slipped through. It now checks only that the cap is stated, as
24 per table or 23 secondary, and keeps the guards against other
units ("24 hours", "24 columns") and upper-bound wording.
Whether the 24 counts the primary key moves to eval 28, an LLM-judge
eval: a PostgreSQL table with a PK, a UNIQUE constraint and 23 other
indexes needs 25 slots in DSQL and does not fit. Eval 3's expectation
returns to its original wording, the PK-inclusive alias is dropped,
and the veto tests go with the veto.
troubleshooting.md loads on any error and had grown to 415 lines. The Limits and Error Codes section moves to limits-and-error-codes.md with its own SKILL.md row, loaded before sizing batches, keys or indexes and on a 54000/0A000/54011 error. troubleshooting.md keeps a pointer and drops to 223 lines. OC001 is now explained once, in occ-retry-patterns.md; the troubleshooting entry keeps the cause, marker table and fix. The 24-slot paragraph in index-conversion.md becomes a link. The SKILL.md OCC and size bullets are shorter but keep the facts evals 24 and 26 showed agents miss without them. Drops the measurement date, the view-size series and "observed here". Also drops the 54001 row and the 214 MiB figure, which did not reproduce on re-test.
The dropped-columns note sent agents to recreate the table, which ends in an irreversible drop. Say it needs user approval, alongside the link to the procedure that carries the full gate.
For limit and error-code questions agents answered from the AWS docs and never loaded the skill, so the measured limits never reached them. The description now asks for the skill on those questions. Where a measured limit contradicts the quotas page, such as key size, the reference now says to size against the measured value and mention the documented one, instead of re-verifying against the docs.
The OCC and transaction-size bullets in Error Scenarios restated the references. They now route to occ-retry-patterns.md and limits-and-error-codes.md, keeping only "keep OC001 in the retry loop" and "size is checked at COMMIT"; eval 24 dropped the COMMIT fact without it. Eval 26 expectation 2 required "not two statements overlapping in time", which fails a correct answer: the prompt fails at COMMIT, and OC001 surfaces at COMMIT when the change lands mid-transaction. It now accepts a GRANT that finished before the transaction or one that committed during it. The expected_output and README row match.
|
Thanks for the review. Addressed the reviews and the claims were re-measured on a cluster.
|
This PR corrects the DSQL skill's error guidance and adds a reference for DSQL's limits and error codes.
Agents use
references/troubleshooting.mdto match a DSQL error and decide what to do next. Five of its quoted messages did not match what DSQL returns, so those errors went unmatched. Common limit errors, such astransaction row limit exceeded, had no entry at all.Changes
references/limits-and-error-codes.md, loaded before sizing batches, keys or indexes and on a limit error: 22 limits, 20 rejections and 10 cluster quotas, each with its SQLSTATE and exact message.54000and0A000each cover more than a dozen rows, so agents match on the message, case-insensitively.Token has expired,Transaction exceeds 3000 rows,Datatype array not supported,Please use CREATE INDEX ASYNC,OC001 - Concurrent DDL operation.OC001is a40001raised by a stale session catalog cache. It hits reads too, so it stays in the retry loop.Signature expiredhint means a stale token.ASC/DESCon index keys, anddsql_lintstrips them, silently forASC.COMMITor on an autocommit write leaves the outcome unknown, so the agent confirms before resending.UNIQUEconstraints.54000DETAIL.COMMIT, catalog change, key size, index count on migration.Follows #278, which flagged the inherited
sort orderline inindex-conversion.mdfor this PR to drop.Verification
Every quoted message was reproduced against a live cluster. Cluster quotas are documented only, not exercised.
mise run buildpasses, including 132 unit tests.Functional evals,
claude-sonnet-5-5as subject and judge, 4 runs per eval with the skill and againstmain. Graded expectations passed:mainCOMMITThese used a local copy of the fix(databases-on-aws): harden DSQL functional eval harness #262 runner with fixes, the main one being its
Readpermission rule. As committed, that rule denies every read of a skill reference file, so a functional eval only ever exercisesSKILL.md. The other fixes accept the skill text block the CLI sends on load, truncate long judge evidence instead of discarding the verdict, allowaws___read_documentation, and handle parallel tool calls. Runs lost to macOS process-containment errors are excluded. Evals 24 and 26 are from the final commit; the others are from the commit before it, which differs only in twoSKILL.mdbullets that evals 24 and 26 test. The eval 26 baseline ran before its expectation 2 was reworded. Eval 25 doesn't separate the skill frommain, because the base model already gets it right.Runner summary per run (
summary.json)with= this branch,base=main.r9ran evals 3 and 23–28. The table's eval 24 is fromr11and its eval 26 fromr12, both at the final commit;r12ran after eval 26 expectation 2 was reworded.Version
1.9.1→1.10.0: new guidance plus an eval change.Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.