Skip to content

docs(dsql): add limits/error-code reference, fix wrong messages - #279

Merged
krokoko merged 8 commits into
mainfrom
fix/dsql-limits-and-error-messages
Oct 9, 2026
Merged

krokoko merged 8 commits into
mainfrom
fix/dsql-limits-and-error-messages

Conversation

@praba2210

@praba2210 praba2210 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

This PR corrects the DSQL skill's error guidance and adds a reference for DSQL's limits and error codes.

Agents use references/troubleshooting.md to match a DSQL error and decide what to do next. Five of its quoted messages did not match what DSQL returns, so those errors went unmatched. Common limit errors, such as transaction row limit exceeded, had no entry at all.

Changes

  • New references/limits-and-error-codes.md, loaded before sizing batches, keys or indexes and on a limit error: 22 limits, 20 rejections and 10 cluster quotas, each with its SQLSTATE and exact message. 54000 and 0A000 each cover more than a dozen rows, so agents match on the message, case-insensitively.
  • Skill description now asks for the skill on DSQL limit and error-code questions. Without it, agents answered those from the AWS docs and never loaded the skill. Where a measured limit contradicts the quotas page, the reference says to size against the measured value.
  • Five wrong messages corrected: Token has expired, Transaction exceeds 3000 rows, Datatype array not supported, Please use CREATE INDEX ASYNC, OC001 - Concurrent DDL operation.
  • Changes:
    • OC001 is a 40001 raised by a stale session catalog cache. It hits reads too, so it stays in the retry loop.
    • Only the Signature expired hint means a stale token.
    • DSQL rejects ASC/DESC on index keys, and dsql_lint strips them, silently for ASC.
    • An EOF on COMMIT or on an autocommit write leaves the outcome unknown, so the agent confirms before resending.
  • Limits:
    • Every index counts toward the 24, including the primary key and UNIQUE constraints.
    • Key size is ~1,981 bytes for one column, about 7 fewer per extra column, and a secondary index shares the budget with the primary key.
    • Secondary indexes count toward the 10 MiB transaction size; agents size batches from the 54000 DETAIL.
  • Evals:
    • Six added: sort order, transaction size, EOF at COMMIT, catalog change, key size, index count on migration.
    • Eval 3's regex now checks only that the 24-index limit is stated. Whether the answer counts the primary key inside the 24 moves to eval 28, an LLM-judge eval, since that turns on paraphrase and negation.

Follows #278, which flagged the inherited sort order line in index-conversion.md for this PR to drop.

Verification

  • Every quoted message was reproduced against a live cluster. Cluster quotas are documented only, not exercised.

  • mise run build passes, including 132 unit tests.

  • Functional evals, claude-sonnet-5-5 as subject and judge, 4 runs per eval with the skill and against main. Graded expectations passed:

    Eval With skill main
    3. Index limits 16/16 not graded: skill never loaded
    23. Index sort order 10/13 1/3
    24. Transaction size 8/9 8/13
    25. EOF at COMMIT 4/4 10/10
    26. Catalog change (OC001) 9/9 3/5
    27. Key size 15/15 not graded: skill never loaded
    28. Index count on migration 9/9 2/5

    These used a local copy of the fix(databases-on-aws): harden DSQL functional eval harness #262 runner with fixes, the main one being its Read permission rule. As committed, that rule denies every read of a skill reference file, so a functional eval only ever exercises SKILL.md. The other fixes accept the skill text block the CLI sends on load, truncate long judge evidence instead of discarding the verdict, allow aws___read_documentation, and handle parallel tool calls. Runs lost to macOS process-containment errors are excluded. Evals 24 and 26 are from the final commit; the others are from the commit before it, which differs only in two SKILL.md bullets that evals 24 and 26 test. The eval 26 baseline ran before its expectation 2 was reworded. Eval 25 doesn't separate the skill from main, because the base model already gets it right.

    Runner summary per run (summary.json)

    with = this branch, base = main. r9 ran evals 3 and 23–28. The table's eval 24 is from r11 and its eval 26 from r12, both at the final commit; r12 ran after eval 26 expectation 2 was reworded.

    r9-with-1  evals=7  requested=33  graded=20  passed=19  failed=1  infra_errors=5
    r9-with-2  evals=7  requested=33  graded=24  passed=24  failed=0  infra_errors=6
    r9-with-3  evals=7  requested=33  graded=19  passed=18  failed=1  infra_errors=5
    r9-with-4  evals=7  requested=33  graded=24  passed=22  failed=2  infra_errors=3
    r9-base-1  evals=7  requested=33  graded=8   passed=5   failed=3  infra_errors=7
    r9-base-2  evals=7  requested=33  graded=13  passed=9   failed=4  infra_errors=6
    r9-base-3  evals=7  requested=33  graded=12  passed=8   failed=4  infra_errors=5
    r9-base-4  evals=7  requested=33  graded=3   passed=2   failed=1  infra_errors=7
    r11-1      evals=2  requested=10  graded=7   passed=7   failed=0  infra_errors=2
    r11-2      evals=2  requested=10  graded=0   passed=0   failed=0  infra_errors=2
    r11-3      evals=2  requested=10  graded=5   passed=5   failed=0  infra_errors=1
    r11-4      evals=2  requested=10  graded=9   passed=7   failed=2  infra_errors=1
    r12-1      evals=1  requested=5   graded=0   passed=0   failed=0  infra_errors=1
    r12-2      evals=1  requested=5   graded=2   passed=2   failed=0  infra_errors=1
    r12-3      evals=1  requested=5   graded=3   passed=3   failed=0  infra_errors=1
    r12-4      evals=1  requested=5   graded=4   passed=4   failed=0  infra_errors=1
    

Version 1.9.1 → 1.10.0: new guidance plus an eval change.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

Copilot AI balanced review requested due to automatic review settings September 30, 2026 22:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@praba2210
praba2210 force-pushed the fix/dsql-limits-and-error-messages branch from d43958d to a136fb4 Compare September 30, 2026 23:41
Copilot AI balanced review requested due to automatic review settings September 30, 2026 23:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:13
@praba2210
praba2210 force-pushed the fix/dsql-limits-and-error-messages branch from a136fb4 to 703c080 Compare October 1, 2026 19:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@praba2210
praba2210 force-pushed the fix/dsql-limits-and-error-messages branch 2 times, most recently from 10a0809 to 924d8e8 Compare October 1, 2026 21:22
Copilot AI balanced review requested due to automatic review settings October 1, 2026 21:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 1, 2026 22:29
@praba2210
praba2210 force-pushed the fix/dsql-limits-and-error-messages branch from 924d8e8 to 2706eca Compare October 1, 2026 22:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@praba2210
praba2210 marked this pull request as ready for review October 5, 2026 17:29
@praba2210
praba2210 requested review from a team as code owners October 5, 2026 17:29
@praba2210

Copy link
Copy Markdown
Contributor Author

Second review pass against a live cluster.

Issue Fix
troubleshooting.md still defined OC001 as "Multiple DDL operations on the same resource" / "Concurrent DDL", contradicting eval 26 and occ-retry-patterns.md in this same PR Snapshot predates any catalog change; one statement is enough and need not be concurrent. A single already-committed ALTER TABLE, GRANT and REVOKE each raise it alone
Nothing stated that OC001 only affects transactions that write — which also made eval 26's read-only reporting session an impossible premise Added to both files; 4/4 read-only variants commit clean, including explicit BEGIN READ ONLY. Eval 26's job now writes
The grader still failed correct answers in two shapes: the singular the 24-index limit, and an answer that also stated truthfully that the primary key does not count toward the 1,600-column limit — the latter reported as the off-by-one it polices Singular accepted; veto now ignores statements about another limit's unit. Test for each
Overturned by the cluster, left as written OC001 is reported at COMMIT to the later committer; DSQL classifies GRANT/REVOKE as DDL itself (0A000: multiple ddl statements not supported in a transaction for a pair)
Out of scope, filing separately GRANT counts against the one-DDL-per-transaction rule, so references/access-control.md showing eight consecutive GRANT statements as one block fails

Also corrected the eval README's prompt and assertion counts and two stale wordings.

mise run build passes — 133 tests, 27 evals, 118 assertions.

@anwesham-lab

Copy link
Copy Markdown
Member

Code review

The corrected error messages, the limits and rejection tables, and the plan-shape guidance hold up on a live cluster. Four behavioral claims did not hold up, and three of them were added or changed in the last two self-review passes: (1) OC001 does hit read-only transactions, (2) OC001 comes from a stale per-session catalog cache, not from a snapshot that predates the change, (3) the ~430 B per index entry overhead and (4) the ~1,955 / ~1,455 key-size figures. The eval-3 grader also passes wrong answers and fails correct ones. Blocking: #1, #2, #4. Every DSQL claim below was re-run against a single-Region PostgreSQL 16 cluster (us-east-1) on 2026-10-06. All test objects used a pr279_rv_ prefix and were dropped afterwards.

# Confidence Area Finding Suggestion Validation Reviewed SHA
1 95 troubleshooting.md#L388-L392, occ-retry-patterns.md#L50-L54 correctness "Only transactions that write are affected … a read-only reporting workload needs no OC001 handling at all" is false. It tells agents to remove retry handling from read paths. Delete the paragraph in both files. Keep 40001 handling on every path, reads included. Cluster. On a session that had already read the table, an ALTER TABLE ADD COLUMN or GRANT committed from a second session, then the first session read again. Result: OC001 in 23 of 24 runs across implicit SELECT, BEGIN READ ONLY and plain BEGIN+SELECT. A read-only transaction whose change landed mid-transaction committed cleanly in 8 of 8 runs. That is the scenario behind the PR's claim, and it does not generalize.
2 85 troubleshooting.md#L369-L372, #L393-L397, occ-retry-patterns.md#L33-L35, evals.json#L321-L327 correctness evals The OC001 mechanism is wrong. The cause is a stale per-session catalog cache, not "the transaction's snapshot predates the change". It can surface on any statement, not only at COMMIT. The working-with-ddl quote the PR cites ("already completed before the affected transaction starts") contradicts the PR's own explanation. Eval 26 then requires the snapshot explanation, so an answer that matches the AWS docs fails. Describe the cause as "the session's cached catalog is older than a committed catalog change". Say it surfaces at the next statement that touches storage, or at COMMIT. Reword eval 26's expectations 1–2 and the README row to match. Cluster. A change landing mid-transaction gave OC001 at COMMIT in 5 of 5 runs, with the INSERT succeeding; this matches the PR. A change committed before BEGIN gave OC001 on the INSERT statement itself (1 of 3 for writes) and on reads (23 of 24, see #1). In a separate run, an idle session's own GRANT was rejected with OC001.
3 75 troubleshooting.md#L74-L92, #L361-L366, SKILL.md#L297, evals.json#L300 correctness The formula raw + rows × (~150 + ~430 per secondary index) did not reproduce, and "accounted size is always above raw, never below" is false. The formula was added in the last pass (e62a465); earlier revisions said the size was not predictable. Drop the fixed ~430 B constant, or state the index key width it was measured with. Say that per-entry cost scales with the index key plus PK width, and that compressible payloads count below raw. Keep the qualitative rule ("secondary indexes shrink the 10 MiB budget") in SKILL.md and state the numbers once. Cluster. 500 rows of random bytea in an unindexed column. With 0 secondary indexes, 9.9 MiB committed (formula predicts 9.93). With 4 valid int secondary indexes, 9.6 MiB committed and 9.8 MiB failed 54000 (formula predicts 9.11 MiB). That puts the cost near 120 B per entry, not 430. Separately, a 50 MiB repeat('a', …) value and 16 MiB of compressible text each committed inside one transaction.
4 95 run_functional_evals.py#L4965-L5079, evals.json#L38-L40 evals The INDEXES_PER_TABLE regex now fails in both directions. Wrong answers pass: "up to 24 columns per index", "can take up to 24 hours", "the primary key is excluded", and "24 secondary indexes … 23 indexes fits" (the 23 elsewhere in the answer cancels the veto). Correct answers fail: "23 secondary indexes plus the primary key", "cannot have more than 24 indexes", a following sentence starting "In addition to the primary key, create…", the curly-apostrophe "can’t exceed", and "a table with no primary key accepts 24 secondary indexes" (true, see #5). The assertion text says "including the primary key", but a bare "at most 24 indexes" passes. Several alternatives the code comment says must be tested have no test. Keep the regex for "states 24 / 23 secondary" only. Move the PK-inclusion check into an llm_judge eval, which is what the README recommends for negation and paraphrase. If the regex stays, scope states_23 per statement, require the secondary qualifier, and add each probe above as a test. Local. Ran grade_eval through the test module's _regex_run_result. All 10 probes above graded as stated. A test-analyzer mutation run found 7 of 18 grader mutations survive the suite, including both per-statement scoping guards.
5 90 troubleshooting.md#L50, development-guide.md#L75, examples/schema.md#L35, SKILL.md#L156-L163 correctness "24 including the primary key, so 23 secondary" is stated without condition. A table with no declared PK accepts 24 secondary indexes. The author's first self-review comment recorded this ("no-PK table 24 max"), but it never reached the docs. Say "24 per table; a declared primary key uses one, leaving 23 secondary". Accept the no-PK case in the grader (#4). Cluster. PK table: index 24 rejected with 54000 more than 24 indexes per table are not allowed. No-PK table: 24 accepted, index 25 rejected with the same error. This confirms the PR's 23 for PK tables.
6 85 troubleshooting.md#L93-L99, evals.json#L335-L339 correctness The "~1,955 combined ceiling" and "a 500-byte primary key leaves ~1,455" are about 19 bytes low. Self-review row 14 changed ~1,474 to ~1,455; the measurement supports the original 1,474. The bullet also treats multi-column keys and secondary-index-plus-PK as two different budgets, but they measure the same. State one rule: about 1,981 bytes minus about 7 per extra key column, and a secondary index key includes the PK columns (about 1,974 for one index column plus a one-column PK). Cluster. Random-text binary search. Single-column PK: 1,981 ✓. Eight-column PK: 241 each ✓. Two-column PK: 1,975 combined. Secondary index with a 10 / 500 / 1,000-byte PK: index column max 1,965 / 1,474 / 974, so 1,974–1,975 combined every time.
7 85 dsql-lint.md#L53-L57, index-conversion.md#L281-L284 correctness Both proposed gates (len(diagnostics) == 0, or fixed_sql equal to the input) pass SQL that DSQL rejects. #278 documented that operator classes produce no diagnostic, and the PR's own Rejections table lists 0A000 opclass not supported for index keys. The fixed_sql comparison also passes unfixable diagnostics when no rewrite happens. The gate also conflicts with #278's BEGIN ATOMIC parse_error exception. Present len(diagnostics) == 0 as necessary, not sufficient, and link the opclass manual check and the BEGIN ATOMIC exception. Drop the fixed_sql alternative. Ran dsql_lint(fix=true) on ASC, DESC and text_pattern_ops. Results: ASC fixed (silent), DESC fixed_with_warning (the PR's severity table is right), opclass 0 diagnostics with fixed_sql unchanged.
8 70 troubleshooting.md#L150-L176, occ-retry-patterns.md#L62-L65 safety The confirm-before-resend MUST only covers an EOF "at or after COMMIT". In autocommit, every write is its own commit, so an EOF there is equally uncertain. "Reduce the batch size until the EOF stops" is an unbounded resend with no check on what landed. occ-retry-patterns.md offers "or make the write idempotent" as an alternative to confirming, while troubleshooting.md says idempotency cannot be applied after the fact. Widen the scope to "any statement that could have committed, including autocommit writes". Make batch reduction conditional on confirming the previous attempt did not commit. State the confirm-or-stop sequence once and link to it from occ-retry-patterns.md. Cluster. An 11 MB autocommit INSERT returned SSL error: unexpected eof, and a fresh session found 0 rows with its marker. That confirms an oversized message never executes. A network EOF on a normal-sized autocommit write has no such guarantee, which is why the rule's scope matters.
9 90 troubleshooting.md#L197-L201 correctness The note says the three creating more than … messages are "the only rows whose capitalisation differs". They are lowercase, like the rows around them. The rows that do differ include Datatype limit greater than 4096/65535 … (observed capitalised) and the Unable …, You have reached … and Current cluster size … quota rows. An agent matching case-sensitively would miss the char/varchar rows. Delete the sentence and say to match case-insensitively throughout. Cluster. char(5000) and varchar(70000) both returned 54000 Datatype limit greater than … with a capital D.
10 70 troubleshooting.md#L111-L114 safety "Recreate the table rather than continuing to churn it" tells an agent to perform a destructive copy, swap and drop with no approval gate. SKILL.md Workflow 7 requires "a user-approved table-recreation plan". Point to ddl-migrations/overview.md and add "MUST present the plan and get user approval before dropping". Read SKILL.md Workflow 7 at head. No cluster test needed.
11 85 SKILL.md#L161-L163, #L294-L297; troubleshooting.md overall size style SKILL.md grows from 299 to 303 lines, past the 300-line limit (DESIGN_GUIDELINES.md; also requested in #168). The OCC and limits bullets explain rather than route. troubleshooting.md grows from 165 to 421 lines and loads on any error. Repeated content: OC001 is explained three times, with one paragraph identical in two files; the size formula appears three times; "24 incl. PK" appears in six places. Lab-notebook detail (MiB series, view sizes by query shape, the 2026-10-01 date, "observed here") costs tokens without changing what the agent does. Move "Limits and Error Codes" into references/limits-and-error-codes.md with its own SKILL.md row and a proactive "When:" condition. Give each rule one home and replace the copies with links. Cut SKILL.md:161–163, :294 and :297 to one-line pointers. Move measurement evidence into the PR. wc -l. tools/validate-size.py marks dsql as REVIEW. Duplicate paragraph confirmed with grep.
12 90 PR body and commit message pr-body commit-msg Factual errors: (a) 54000 covers 16 rows, not 15. (b) #278 did not say dsql_lint preserves sort order; that text dates to #168, and #278 flagged it for this PR to fix. (c) "Six tests pin 20 phrasings" is stale; the commit body is byte-identical to e62a465, so it predates the last pass. (d) The body gives ~1,981 bytes as shared with secondary indexes, but the file says ~1,955 (and #6 shows both are off). Correct (a)–(d). Mention the read-only OC001 claim and the grader scoping changes from the last pass. Counted rows at head. gh pr view 278 body says "#279 drops it". Diffed the commit bodies of e62a465 and ebbd8a2: empty diff.
13 80 PR body "Verification" evals "mise run build passes — … 27 functional evals, 118 assertions" reads as an eval run. The build runs only pytest (mise.toml); 27 and 118 are counts in evals.json. No functional run, and no with-skill vs. baseline comparison, is shown for evals 3 or 23–27. #1–#3 show runtime claims shipped without being exercised. Attach run_functional_evals.py results (pass/fail per expectation, subject and judge model) for evals 3 and 23–27, plus a baseline, after fixing #1–#6. Read mise.toml build and test tasks. Searched the PR and both self-review comments for eval artifacts: none found.
Findings dropped during validation (21)
Candidate (source agent) Why dropped
OC001 is reported on the statement, per the docs example, not at COMMIT (DSQL-claims, code-reviewer) Refuted for the PR's scenario. When the change landed mid-transaction, the INSERT succeeded and OC001 came at COMMIT in 5 of 5 runs. The true nuance is now part of #2.
Restore #262's "24 secondary indexes" expectation, since PK inclusion is undocumented (git-history) Refuted. The cluster rejects the 24th secondary index on a PK table. The no-PK nuance is now #5.
char/varchar limits are checked at write time, not DDL time (DSQL-claims) Refuted. CREATE TABLE itself failed with 54000.
The backward-scan / plan-table claims overstate planner choice (past-PR-comments) Refuted. EXPLAIN on 20k rows reproduced all four shapes: Index Only Scan Backward, Incremental Sort, Sort for SELECT *, and Sort for a non-prefix order.
dsql_lint ASC/DESC severities are unverified (several) Confirmed correct by running dsql_lint (#7).
256 vs 255 columns, and the column-count messages (DSQL-claims) Confirmed correct. 256 accepted and 257 rejected, with or without a PK, with the messages as quoted.
GRANT pair rejected as multiple DDL (several) Confirmed correct. Returned 0A000 multiple ddl statements not supported in a transaction.
GRANT/REVOKE raise OC001 (several) Confirmed correct.
Rejection messages: sort order, no-ASYNC, text[], NULLS FIRST accepted (body-quality) Confirmed verbatim.
Wrappers drop HINT, so "no hint means credentials" misclassifies (silent-failure) Confirmed the PR's rule. The expired-token run returned HINT: Signature expired; the wrong-Region run returned a different hint. Wrappers dropping HINT is speculation.
OC001 on SELECT … FOR UPDATE (git-history, silent-failure) Moot. Plain reads already hit OC001 (#1).
Regex catastrophic backtracking in the grader (security) Refuted. The security agent ran 17–36 KB adversarial inputs and each finished in about 0.01 s.
SKILL.md "Token expiration: generate a fresh token" left unscoped (superpowers) Unmodified line, already scoped to expiration.
"40001 is the only retryable code" vs the connection table's "Retry: yes" (several) Customer-value gate: the sentence is scoped to these lookup tables, so a reader is unlikely to be misled. Nit.
marker defaults to OC000 when no marker is present (silent-failure, language) Every 40001 observed on the cluster carried a marker. Low value.
rollback() on a closed connection masks the EOF in the retry sample (language, silent-failure) Pre-existing lines, not modified. The safety concern is covered by #8.
UPDATE accounts SET balance = balance - 100 has no WHERE (security) An illustrative fragment in prose, not a runnable example. Low value.
The PK does not count toward the 1,600-column ceiling (comment-analyzer, test docstring) Not run on the cluster (needs about 1,600 DDLs). Test-docstring wording only.
Bump GRADING_PROTOCOL_VERSION; unsigned commit; stale author date; docs vs fix type; open an RFC or issue (AGENTS.md, commit audit) No MUST rule applies. A squash merge re-signs and re-dates the commit. CI passes.
View size: the table's 131 KiB lower bound vs the ~127 KiB rejection in the bullet (drift) Nit. The ~120 KiB working ceiling already covers it.
Quote "regardless of the number of secondary indexes" not found in the docs (past-PR-comments, DSQL-claims) Under 60 after validation: two searches failed to find it, but the docs MCP read tool was broken, so it could not be ruled out. Worth the author citing the source.
Self-review passes and drift

Eight force-pushed revisions; the base has not moved (merge-base da51970 = main). Pass 1 has 17 rows and pass 2 has 5. Every claimed fix is present at head. Results by row:

Self-review item Present Holds on cluster / in code
P1 #1–2: grader accepts more answer shapes; veto widened Yes Partly. It now fails correct answers and passes wrong ones (#4).
P1 #3–4, #13: ASC/DESC severity; NULLS retained; lint gate Yes Severities correct. Gate unsafe (#7).
P1 #5: plan shapes over a leading prefix Yes Correct (EXPLAIN).
P1 #6: replace "not predictable" with the formula Yes Does not reproduce (#3).
P1 #7, P2 #1: OC001 = any catalog change, including GRANT/REVOKE Yes Trigger correct. Mechanism wrong (#2).
P1 #11: 24 includes the PK Yes Correct for PK tables; the no-PK case was dropped (#5).
P1 #14: key budget changed from ~1,474 to ~1,455 Yes Regression. 1,474 was right (#6).
P1 #15: MUST NOT retry when the outcome is unknown Yes Correct, but scoped too narrowly (#8).
P2 #2: only writing transactions hit OC001 Yes False (#1).
P2 #3: grader accepts the singular form; ignores other limits Yes Partly; the 8-column-per-index case still fails.
P2 "filing separately": access-control.md GRANT block n/a No issue filed as of 2026-10-06. The block has 7 GRANTs plus 7 other DDL statements, not "eight consecutive GRANTs".

Drift: the claims that changed most across revisions are the ones that fail on the cluster. OC001 semantics changed four times, the read-only exemption was added at the last head, the 10 MiB model flipped from "unpredictable" to a formula at e62a465, and key size changed four times. Each pass fixed the contradictions the previous one introduced (e62a465 still said "concurrent DDL" in troubleshooting.md), but the passes were checked against themselves, not re-measured. Assessment: structure and direction are sound, and the messages, tables and plan guidance are good. This needs one more targeted pass, not a rewrite. Fix #1–#6 against a cluster, move the PK-inclusion check to the judge, split the limits section out to bring SKILL.md back under 300 lines, and attach a real functional-eval run.

Reviewed by 21 sub-agents: code-review 4a–4e, silent-failure-hunter, comment-analyzer, code-simplifier (report-only), pr-test-analyzer, type-design-analyzer, pr-review-toolkit code-reviewer, superpowers code-reviewer, security, drift, PR-body, language-idiomatic, authoring-style, AGENTS.md compliance, prior-comment resolution, commit audit, DSQL docs-claims. Findings were consolidated after all agents returned, then validated against the head SHA, a live DSQL cluster, dsql_lint, and local grader probes.

🤖 Generated with Claude Code

Comment thread plugins/databases-on-aws/skills/dsql/references/auth/authentication-guide.md Outdated
Comment thread plugins/databases-on-aws/skills/dsql/references/examples/schema.md Outdated
Comment thread tools/evals/databases-on-aws/dsql/scripts/run_functional_evals.py Outdated
Comment thread plugins/databases-on-aws/skills/dsql/references/pg-migrations/multi-region.md Outdated
Comment thread plugins/databases-on-aws/skills/dsql/references/development-guide.md Outdated
OC001 hits reads too, so drop the read-only exemption. The cause is a
stale per-session catalog cache, not a transaction snapshot; eval 26
now accepts the documented completed-before-start case.

Drop the 10 MiB formula. Per-entry cost depends on index key and PK
width, so point agents at the 54000 DETAIL instead.

Every index counts toward the 24, UNIQUE included. Key budget is
~1,981 bytes less ~7 per extra column, 1,474 with a 500-byte PK. Add
the 2,000-byte DDL-time key check.

Zero lint diagnostics is not acceptance; drop the fixed_sql compare.
The EOF rule now covers autocommit writes. Fix the capitalisation
note and trim SKILL.md to 299 lines.

Fix INDEXES_PER_TABLE misgrades in both directions and pin each
probe in tests.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 05:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The INDEXES_PER_TABLE grader still misgraded close variants of the
review probes. "up to 24 hours" after an index noun passed as the
cap, and correct answers failed: one naming DSQL rather than the
table, the no-primary-key caveat on its own (its "no" read as a
negation), and "23 secondary plus the primary key, 24 in total".
Each now grades correctly; five new test strings cover them.

occ-retry-patterns.md now links to the confirm-or-stop rule in
troubleshooting.md instead of offering idempotency as an alternative
to confirming. development-guide.md trims the index limit to one
clause.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 18:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The INDEXES_PER_TABLE regex tried to veto every phrasing that puts
the primary key outside the 24, and each probe round found another
that slipped through. It now checks only that the cap is stated, as
24 per table or 23 secondary, and keeps the guards against other
units ("24 hours", "24 columns") and upper-bound wording.

Whether the 24 counts the primary key moves to eval 28, an LLM-judge
eval: a PostgreSQL table with a PK, a UNIQUE constraint and 23 other
indexes needs 25 slots in DSQL and does not fit. Eval 3's expectation
returns to its original wording, the PK-inclusive alias is dropped,
and the veto tests go with the veto.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 19:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

troubleshooting.md loads on any error and had grown to 415 lines.
The Limits and Error Codes section moves to limits-and-error-codes.md
with its own SKILL.md row, loaded before sizing batches, keys or
indexes and on a 54000/0A000/54011 error. troubleshooting.md keeps a
pointer and drops to 223 lines.

OC001 is now explained once, in occ-retry-patterns.md; the
troubleshooting entry keeps the cause, marker table and fix. The
24-slot paragraph in index-conversion.md becomes a link. The SKILL.md
OCC and size bullets are shorter but keep the facts evals 24 and 26
showed agents miss without them.

Drops the measurement date, the view-size series and "observed
here". Also drops the 54001 row and the 214 MiB figure, which did
not reproduce on re-test.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 20:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The dropped-columns note sent agents to recreate the table, which ends
in an irreversible drop. Say it needs user approval, alongside the
link to the procedure that carries the full gate.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 20:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Prabakaran Annadurai added 2 commits October 8, 2026 16:07
For limit and error-code questions agents answered from the AWS docs
and never loaded the skill, so the measured limits never reached
them. The description now asks for the skill on those questions.

Where a measured limit contradicts the quotas page, such as key size,
the reference now says to size against the measured value and mention
the documented one, instead of re-verifying against the docs.
The OCC and transaction-size bullets in Error Scenarios restated the
references. They now route to occ-retry-patterns.md and
limits-and-error-codes.md, keeping only "keep OC001 in the retry
loop" and "size is checked at COMMIT"; eval 24 dropped the
COMMIT fact without it.

Eval 26 expectation 2 required "not two statements overlapping in
time", which fails a correct answer: the prompt fails at COMMIT, and
OC001 surfaces at COMMIT when the change lands mid-transaction. It
now accepts a GRANT that finished before the transaction or one that
committed during it. The expected_output and README row match.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 23:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@praba2210

praba2210 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Addressed the reviews and the claims were re-measured on a cluster.

# Status Change
1 Fixed Read-only paragraph removed; reads need the retry loop.
2 Fixed The cause is a stale session catalog cache. It surfaces on the statement, or at COMMIT when the change lands mid-transaction. Eval 26 and its README row are reworded.
3 Fixed Formula removed. The rule stays, with one measured example and a pointer to the 54000 DETAIL.
4 Fixed Eval 3's regex only checks that the limit is stated. The PK check moved to a new LLM-judge eval 28.
5 Fixed Every index counts, the PK and UNIQUE included. A table with no PK holds 24.
6 Fixed One rule: ~1,981 bytes, about 7 fewer per extra column, ~1,474 beside a 500-byte PK.
7 Fixed Zero diagnostics means "no lint rule fired", not acceptance. The fixed_sql comparison is removed.
8 Fixed Covers autocommit writes. Batch reduction requires confirming first. occ-retry-patterns.md links to the rule.
9 Fixed Sentence removed; matching is case-insensitive.
10 Fixed The sentence now says to recreate the table with user approval, and links to ddl-migrations/overview.md, which carries the full gate.
11 Fixed
  • Limits moved to limits-and-error-codes.md, and troubleshooting.md went from 415 to 223 lines.
  • OC001 has one home, and the duplicates are now links.
  • Lab-notebook detail is removed, along with the unverified 54001 row and the 214 MiB figure.
  • The SKILL.md OCC and size bullets are now one-line pointers. Each keeps one clause, "keep (OC001) in the loop" and "size is checked at COMMIT": eval 24 dropped the COMMIT fact without it.
12 Fixed PR body corrected. A clean squash message is drafted for merge.
13 Fixed Graded results for evals 3 and 23–28, with the skill and against main, are in the PR body. Getting them needed local fixes to the #262 runner: as committed, it denies every read of a skill reference file, so functional evals only exercise SKILL.md.

authentication-guide.md, schema.md and development-guide.md are cut to one line, using your suggestion for schema.md, and the grader comment is in active voice. On development-guide.md:75, eval 28 shows the baseline hedging on whether the PK counts, since the quotas page doesn't say. In multi-region.md, the link is gone and only the "per Region" correction from the quotas page remains.

@praba2210
praba2210 requested a review from anwesham-lab October 9, 2026 00:17

@anwesham-lab anwesham-lab left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, ty!

@krokoko
krokoko added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit fbefb37 Oct 9, 2026
36 checks passed
@krokoko
krokoko deleted the fix/dsql-limits-and-error-messages branch October 9, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants