[arthurschreiber]

The token parser used to pull data from the message through an async iterator. That turned every token into promises and generator yields, and wrapped the output in a Readable stream for backpressure.

This PR turns it into a push parser: data is written in, and tokens are read out synchronously. The public API doesn't change.

Changes

• Push API (src/token/stream-parser.ts):
  • write(chunk) adds data.
  • read() returns the next token, or undefined when it needs more data.
  • end() throws if the data stopped partway through a token.
• Tokens are still parsed lazily, one at a time. Handling a token can change how the tokens after it are parsed. For example, LOGINACK sets the TDS version.
• Short tokens are parsed again from their start once enough data has arrived.
  • The parser tracks how many bytes the current token still needs, and holds incoming chunks until that many are there.
  • So a large token spread over many chunks isn't re-parsed, and re-copied, for every chunk.
• Tokens that can grow arbitrarily large (COLMETADATA, ROW, NBCROW and RETURNVALUE) are parsed incrementally by TokenReaders. These keep their progress, column by column, across chunks.
• max (PLP) values are read by a PLPReader, which consumes whatever part of the value is available instead of waiting for whole PLP chunks. Reading a large PLP chunk now takes time proportional to its size, where it used to grow quadratically.
• token-stream-parser.ts now feeds the message into the push parser and hands tokens to the handler, pausing between tokens when asked to.
  • It emits end and error via process.nextTick, outside the promise that drives parsing.
  • So an exception thrown by a listener, such as a request's completion callback, still surfaces as an uncaught exception, as on master, rather than an unhandled promise rejection.
• Removed: the unused metadataParse from metadata-parser.ts.
• Benchmark fix: benchmarks/token-parser/simple-tokens.js crashed on master with debug.token is not a function. It passed {} as the parser's debug logger; it now passes a no-op token() stub, like the other token-parser benchmarks.

Behaviour change

All tokens from one packet are now handled in one go. Promise callbacks that token event listeners (such as row) queue therefore run after the whole packet, instead of between tokens. Synchronous pause() and cancel() calls in a listener still take effect before the next token.

Performance

Both builds (master at e3f43f7 and this branch) ran on the same machine against SQL Server 2022 in Docker. Each configuration ran twice and the better run is shown; higher is better.

Token parser microbenchmarks (benchmarks/token-parser/, iterations per second):

Benchmark	master	branch
simple-tokens n=100	2,800	3,525	1.26×
simple-tokens n=1000	6,937	10,963	1.58×
done-token tokenCount=10 n=1000	60,810	107,100	1.76×
done-token tokenCount=1000 n=1000	4,600	16,215	3.53×
done-token tokenCount=10000 n=1000	461	2,150	4.66×
colmetadata-token tokenCount=10 n=1000	28,696	34,431	1.20×
colmetadata-token tokenCount=1000 n=1000	1,244	1,854	1.49×
colmetadata-token tokenCount=10000 n=1000	132	187	1.42×

Queries against SQL Server (benchmarks/query/, queries per second):

Benchmark	master	branch
select-many-rows size=100 n=1000	733	791	1.08×
select-many-rows size=1000 n=1000	323	469	1.45×
select-many-rows size=10000 n=1000	55	86	1.58×
select-nvarchar size=10 n=1000	902	965	1.07×
select-nvarchar size=10000 n=1000	795	787	0.99×
select-varbinary size=100 n=100	748	911	1.22×
select-varbinary size=1048576 n=100	161	151	0.94×
select-varbinary size=10485760 n=100	21.0	20.8	0.99×

Across the full result matrix, the token-parser benchmarks range from 0.98× to 4.66×. The query benchmarks range from 0.93× to 1.58×. The largest gains are on result sets with many rows. Single large varbinary(max) values stay within run-to-run noise (0.93–1.15×), where the network transfer dominates.

Tests

• The existing token parser unit tests are adapted to the push API.
• A new test/unit/token/stream-parser-test.ts covers:
  • read() returning undefined until a token is complete;
  • lazy parsing: an option change applies to the tokens after it;
  • end() throwing on data that stops partway through a token or a partly parsed row;
  • ROW and RETURNVALUE tokens with PLP values, and NBCROW tokens, arriving one byte at a time;
  • PLP null values;
  • rejecting PLP values whose length doesn't match the announced total.
• test/unit/token/token-stream-parser-test.ts checks that exceptions thrown by end and error listeners surface as uncaught exceptions.
• Differential check (outside the test suite):
  • 10 responses were recorded from SQL Server 2022, covering login, all data types, NBCROW, 20k-row results, a 5.9 MB max value, output parameters, and PRINT/RAISERROR.
  • Each was parsed by master's parser and by this branch's, under about 29 chunkings per message, including byte by byte, fixed sizes and random splits.
  • The resulting token streams were identical in every case.

Checks

• npm run lint (eslint and tsc)
• npm run build
• Unit tests: 546 passing
• Integration tests: 468 passing, 1 failing. The failure is Initiate Connect Test, which also fails on master in the local test environment.
• commitlint was not run: it isn't installed in the dev dependencies. The commit messages follow the conventional format.

This is the first of a series of PRs toward a pull-based API for consuming responses, including streaming large values. It stands on its own, without any of them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HGJsu1u44CHnt4nkkQ5ias

[codecov]

Codecov Report

❌ Patch coverage is 97.91667% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.38%. Comparing base (e3f43f7) to head (68a8a6b).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/token/colmetadata-token-parser.ts	90.90%	0 Missing and 1 partial ⚠️
src/token/nbcrow-token-parser.ts	90.90%	0 Missing and 1 partial ⚠️
src/token/returnvalue-token-parser.ts	95.00%	0 Missing and 1 partial ⚠️
src/value-parser.ts	97.14%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1794      +/-   ##
==========================================
+ Coverage   82.50%   84.38%   +1.88%     
==========================================
  Files          95       95              
  Lines        4863     4701     -162     
  Branches      914      880      -34     
==========================================
- Hits         4012     3967      -45     
+ Misses        575      473     -102     
+ Partials      276      261      -15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[arthurschreiber]

Follow-up (not part of this PR): cheaper handling of tokens that cross a chunk boundary

When a token straddles two incoming chunks, the push parser's appendPending concatenates all unconsumed bytes with the next chunk (Buffer.concat). For result sets with many small rows, almost every 4 KB TDS packet ends mid-token, so nearly every packet gets copied into a newly allocated buffer. A profile on the recorded server traffic showed that handling 4 KB chunks costs about 20% compared with parsing one contiguous buffer.

Proposed approach: "stitch" instead of concat

• Keep parsing directly from the incoming chunk.
• When a token crosses the boundary, copy only the leftover bytes plus the start of the next chunk into a small, newly allocated buffer, finish that token there, then continue in the next chunk itself.
• Incoming chunks are never modified, so all existing views into them stay valid: binary values from readBinary/readChars, the NBCROW null bitmap, PLPReader pieces, and later zero-copy streaming of max values.

Alternatives considered

• bl: much slower. Every read goes through slice(), which allocates a Buffer and scans the chunk list. In a microbenchmark it was 7–18× slower than Buffer + concat; see the table below.
• Shifting bytes inside one reused buffer (copyWithin, reallocating only to grow): about as fast as stitching. But it overwrites bytes that other code still holds views to: binary column values returned to users, the NBCROW null bitmap, PLPReader pieces, and streamed max-value chunks. Every one of those would need a copy, it copies every incoming byte, and the buffer never shrinks after a large value.

Microbenchmark: 20,000 rows of two int columns, split into chunks. Each approach ran in its own process; figures are medians of 7 × 50 iterations, and runs vary by about ±5%. It measures only the read loop, not the real parser.

Approach	4 KB chunks, 3 separate runs
Buffer.concat (this PR)	412 / 442 / 425 MB/s
Shift in one reused buffer	518 / 530 / 478 MB/s
Stitch	522 / 539 / 495 MB/s
No chunking (upper bound)	552 MB/s
bl (measured separately, not comparable with the rows above)	about 25 MB/s, vs 177 MB/s for Buffer.concat in the same setup

Prototype in the real parser: stitching gave about +13% on small-row results and was mostly within noise elsewhere. It produced identical tokens to master's parser under about 29 different chunkings of 10 recorded SQL Server 2022 responses. Pairing it with throwing NotEnoughDataError from a separate helper function, so V8 keeps inlining the readInt32LE-style helpers after the first throw, looked promising (about +12% on small rows at 4 KB chunks) but was noisy. Both need proper measurement before they go into a PR.

Other measured follow-ups, also deferred:

• a cheaper check for whether debug output is enabled (+20% on small rows);
• a fast path for max values that are null or arrive in one piece (+34% on nvarchar(max) rows);
• skipping iconv for ASCII varchar data (+12% on a mixed-type query).

---

Generated by Claude Code

[github-actions]

🎉 This PR is included in version 20.3.3 🎉

The release is available on:

• npm package (@next dist-tag)
• GitHub release

Your semantic-release bot 📦🚀