Lite's Job History reads only the selected time range, and tab loads read off the UI thread (#5457) by erikdarlingdata · Pull Request #5553 · erikdarlingdata/PerformanceMonitor

erikdarlingdata · 2026-10-08T10:24:05Z

Why

A user with 30 servers and an 11 GB data folder saw Lite 3.9 hang on the first tab switch. The log showed the Job History read running out of DuckDB's 1 GB memory cap, and the store write lock timing out.

Two causes are fixed here.

The Job History read loaded and sorted each server's whole archived job history before it applied the time range.
A few tab loads awaited their database read on the window's own thread. There the read waits on the store lock, then runs the whole query.

v_job_history removes duplicate rows from the live table and the archive with QUALIFY ROW_NUMBER() OVER (PARTITION BY server_id, instance_id ...). DuckDB moves a filter below that window only when the filter reads nothing but the PARTITION BY columns. server_id = $2 went below it. run_datetime >= $1 did not. So each read loaded and sorted the server's whole job history from the archive, message text included, before the 4-hour filter dropped any rows. The tab makes this read twice per server, once for each of the 30 servers in turn.

Plan, measured on DuckDB 1.5.5 (the version Lite ships):

Before: FILTER (run_datetime >= ...) above WINDOW. The table and parquet scans carry only server_id=2.
After: no filter above the dedup. Both scans carry server_id=2 and run_datetime>=..., so parquet row groups outside the window are skipped.

What changes

Lite/Database/DuckDbInitializer.cs, ArchiveViewDedupKeys (the first commit on this branch): the job_history key is now server_id, instance_id, run_datetime. The same copies still collapse. The collector works out run_datetime on the server. It uses the msdb row's own run_date and run_time, which never change for an instance_id. So every copy of a run has the same value. Rows with a NULL run_datetime still group together, because PARTITION BY treats NULLs as equal.
Five tab and window reads that were awaited on the UI thread now run in Task.Run, as Job History and every ServerTab refresh already did:
- Lite/Controls/AvailabilityGroupsTab.xaml.cs LoadAsync (GetAgTopologyAsync). This runs on the tab's load and refresh.
- Lite/Controls/FinOpsTab.xaml.cs LoadDatabaseSizesAsync (GetLatestServerPropertiesAsync). This runs on the tab's load.
- Lite/Controls/FinOpsTab.xaml.cs RunIndexAnalysis_Click and the plan-navigation callback. Both are clicks. The selected server is read on the UI thread first and the read runs off it.
- Lite/Controls/ServerTab.Plans.cs OpenPlanTab (plan viewer metadata). This is a click.
- Lite/MainWindow.xaml.cs LoadServerTagsAsync. This runs on every Overview refresh, which makes it the most frequent of the five. Both tag reads now go in one hop.
Lite/Windows (the windows, added after the tabs): every store read and write a window starts from a click or its Loaded handler now runs in Task.Run, with control values copied to locals first and results marshalled back by the await, so exceptions reach the same catch and message as before. ManageTagsWindow (the two tag reads, the assign and unassign writes, create, rename, colour and delete), DatabaseStateOverridesWindow (expectations, collector run history, engine edition, reset and save), ProcedureHistoryWindow, QueryStatsHistoryWindow and QueryStoreHistoryWindow (history read, chart rows, cached plan, plan-analysis metadata), WaitDrillDownWindow (the three loads and the plan metadata) and CollectionLogWindow (the drill-down read). Not changed: FetchQueryPlanOnDemandAsync and the other Fetch*OnDemand calls (they go to the monitored SQL Server, not the store), ManageMuteRulesWindow (a JSON file, no store), SettingsWindow (another lane's file; it has no data-service call).
Lite.Tests/TabLoadsOffUiThreadTests.cs (new): a census over Lite/Controls/*.cs, Lite/Windows/*.cs and Lite/MainWindow*.cs. It fails if a dataService.XAsync( call (or EngineEditionAsync(dataService, the one static helper that reads on the caller's thread) is not inside a Task.Run( argument list. The main window's six tag-write names stay exempt there only. A window under Lite/Windows has no exemption. Comments and string literals are blanked first so a parenthesis in them cannot unbalance the scan.
Lite.Tests/ArchiveViewDedupTests.cs: two new tests (below).

Measured (synthetic store, scratch folder only, DuckDB 1.5.5)

The store had 30 servers and job history only. It held 9.8 million rows and took 0.8 GB on disk: 3 monthly archive files with 2,048-row row groups, 8 days of live rows, and 6 hours of copies of live rows in the archive. It took 28 s to build. The read is the one-server, 4-hour Job History SQL (the job_stats and base shape the tab runs). One read at a time, nothing else running.

Settings	Old key	New key
1 GB, 32 threads (3.9's cap, every core)	7.6 s, peak working set 727 MB	0.7 s, 286 MB
2 GB, 8 threads (dev's settings, #5381)	4.3 s, 616 MB (repeat: 4.3 s, 596 MB)	0.5 s, 267 MB (repeat: 0.6 s, 267 MB)

The first lane measured 11.1 s and 718 MB for the old key at 1 GB and 32 threads on the same store. My run measured 7.6 s and 727 MB. The memory agrees, and I do not know why the time differs. The old key needed 600 to 730 MB for one server on this 0.8 GB store, and its memory grows with the server's archive. I did not build a store as large as the reporter's, so I did not see a read go past 1 GB. The 3.2 GB store from the first lane was deleted unmeasured.

Row contents: the 4-hour read for one server returned 556 rows under both keys and at both settings. The row text, ordered over every column and hashed with md5, is 58d7017f83 for all four runs. A second hash leaves out job_history_id, collection_time and message. Those columns depend on which copy of a run the dedup keeps. It is 32839ff535 for all four. Repeat runs gave the same values, so the hash is stable. The first lane's hash was not, because it sorted unordered Python objects.

Both scratch stores are deleted. store was 765 MB, and store4x was already gone.

What the first tab switch waits on

From the code. I did not reproduce the freeze, and no UI was run.

OpenConnectionAsync takes the process-wide ReaderWriterLockSlim read lock with an untimed EnterReadLock() before its first await. DuckDB.NET's async methods complete synchronously (DuckDbLockModelTests.DuckDbAsyncStillCompletesOnTheCallingThread pins this). So a data call awaited from the UI thread waits on the lock there, then runs the whole query there.
Readers do not block each other. A parked writer does block them: while archival, compaction or a CHECKPOINT waits for the write lock, ReaderWriterLockSlim holds back every new reader. The reporter's "could not acquire the store write lock within 5s" line is that writer. The old Job History load ran 30 servers in turn at 4 to 11 s each. It held read locks for minutes. A parked writer queued behind it, and everything new queued behind the writer.
A UI thread blocked in a managed lock wait still pumps Windows messages, which fits a frozen window that Windows does not mark "Not Responding".
Every other data call in Lite/Controls already read inside Task.Run, and the census now checks it. The calls that did not are the five above (the Overview tag refresh runs on every Overview refresh). So the most likely first-switch wait is the queue behind the long Job History reads. The UI-thread calls make it freeze the window instead of waiting in the background.

Calls in Lite/Controls, Lite/Windows and Lite/MainWindow*.cs that sat outside a lexical Task.Run before this PR:

Site	On a tab load?	Result
`AvailabilityGroupsTab.LoadAsync`	yes	fixed
`FinOpsTab.LoadDatabaseSizesAsync`	yes	fixed
`FinOpsTab.RunIndexAnalysis_Click`, plan navigation	click	fixed
`ServerTab.OpenPlanTab`	click	fixed
`MainWindow.LoadServerTagsAsync`	every Overview refresh	fixed
`AlertsHistoryTab.ReadCollectedClocksAsync`, `ServerTab.SafeDiscontinuitiesAsync`	yes	already off the UI thread: only called from inside `Task.Run`. The census allows these two by name and checks that every reference to each helper is inside a `Task.Run`.
the store reads and writes in 7 windows under `Lite/Windows`	click or window load	fixed, writes included.
tag clear in `MainWindow.RemoveServerAsync`	click	not changed. They take `OpenWriteConnectionAsync`, which waits for the write lock at most `WriteLockBudget` (5 s) and then throws. The census lists the six method names.

Why Task.Run and not a change to the lock wait:

ReaderWriterLockSlim is thread-affine: the thread that enters must exit. Moving only the wait off the UI thread inside OpenConnectionAsync fails. The caller resumes on the UI thread, and the release there throws SynchronizationLockException (see LockReleaser, Lite has two conventions for locking a DuckDB write, and one of them is written down as the rule #2463).
A lock that is not thread-affine is a change to the primitive under every caller.
A lock that never blocked still leaves DuckDB's synchronous query on the UI thread.
Task.Run(() => dataService.XAsync()) covers every caller with five small edits, and keeps the enter and exit on one pool thread because DuckDB's async completes synchronously.

Test plan

Not done

Nothing left under Lite/Windows: the store calls the first two commits left are moved and the census covers the folder. SettingsWindow belongs to the memory-setting commits.
The 4x store (3.2 GB) was not measured.

The reporter reply is drafted (private) at handoffs/h2/5457-reply-draft.md.

CHANGELOG

SECTION: Fixed
ENTRY: - Lite's Job History tab reads only the selected time range, and tab loads no longer wait on the database from the window ([#5553]) - Every Job History read loaded and sorted each server's whole archived job history, message text included. Only then did it drop the runs outside the selected range. On a 30-server test store that cost 4 to 8 seconds and 600 to 730 MB per server. On a larger store it can run out of DuckDB's memory cap. The archive view now applies the time range first: 0.5 to 0.7 seconds, under 300 MB, same rows. The Availability Groups and FinOps tab loads, opening a plan viewer from a tab, the Overview's tag refresh, and the Manage Tags, Database State Overrides, query and procedure history, wait drill-down and collection log windows also awaited their database reads and writes on the window's own thread. There they can wait behind archival or a long read. They now read off that thread.
REF:
[#5553]: #5553

DuckDB memory setting

Erik's ruling 10-08: users could not change DuckDB's memory limit, so it is now a Lite setting.

What changed:

New "DuckDB memory limit" box under Dashboard Defaults in Settings, stored as duckdb_memory_limit_gb in settings.json. Default 2 GB (dev's Lite: a read that includes query text can run out of memory at the 1 GB DuckDB limit #5381 value, not lowered). Range: a whole number of GB, 1 GB minimum, maximum 80% of physical memory rounded down, never below 1 GB. A missing, unparsable or out-of-range stored value falls back to 2 GB and logs one line. The Settings window rejects an out-of-range entry with a message that states the range; the rest of the page still saves, and the save dialog says the new limit applies after Lite restarts.
DuckDbInitializer.MainConnectionMemoryLimit is now the setting (read once at startup, before the first connection). ConnectionString, the trim cycle's restore (it parses the connection string, so it follows) and ArchiveService's resting value all follow it. The COPY raise is the larger of 4 GB and the setting. Compaction's own 4 GB instance and the importer's plain connection are unchanged.
A DuckDB out-of-memory error shown in a tab status line, the live snapshot line, the plan retrieval dialogs, the three history windows, the wait drill-down, the collection log window and the FinOps index analysis now ends with a sentence naming "DuckDB memory limit" in Settings and its current value, and logs one warning line with the same text. Detection is by message text (Out of Memory, could not free up enough memory, memory_limit), including inner exceptions.
AppLogger appends the same hint, once, to any WARN or ERROR log line that holds DuckDB's exact Out of Memory Error: text (in the message, or anywhere in the exception chain), so the catch blocks that only log the raw message name the setting too. SQL Server memory errors and lines that already hold the hint are left alone. 8 new tests in AppLoggerDuckDbOutOfMemoryHintTests (11 cases) fail with the hook removed and pass with it.
Threads follow the setting: min(8, cores, limit / 256 MB), at least 1. At the 2 GB default that is 8 threads, as today; the 1 GB minimum runs 4. The trim cycle only moves memory_limit (to 64 MB and back) and the COPY raise only moves memory_limit (to the larger of 4 GB and the setting and back), so neither changes threads; a test pins both. preserve_insertion_order stays on for the main connection (pinned); ParquetCompaction's own setting is untouched.

Why the cap of 8 stays and 256 MB per thread: timed on the laptop's real Lite archive (copied to scratch; the live store file was locked by the running app, so the live part of the store was an empty fresh database and only the archive's parquet files were read; copy deleted afterwards). 32 cores, one process per setting, Job History over 20 weeks (17,812 rows) and the top-queries-by-CPU read over 90 days for the busiest server, first and second run each, peak is the process's peak working set (baseline about 150 MB):

memory	threads	Job History ms	top queries ms	peak MB
1 GB	4	464 / 365 and 392 / 365	262 / 241 and 303 / 292	207 and 205
1 GB	8	412 / 451 and 396 / 365	254 / 233 and 244 / 226	235 and 231
1 GB	32	572 / 507 and 491 / 381	601 / 1093 and 467 / 708	392 and 338
4 GB	8	391 / 342 and 398 / 365	223 / 218 and 260 / 249	229 and 234
4 GB	16	445 / 441 and 437 / 395	288 / 273 and 310 / 316	274 and 276

4 and 8 threads are within noise on the Job History read and 8 is a little faster on the wide read; 16 and 32 are slower and use more memory. So more threads than 8 do not help, the cap stays, and raising the memory setting above 2 GB does not raise threads. The 1 GB minimum keeping 4 threads is the memory-per-thread rule (256 MB is double DuckDB's documented 125 MB minimum). This store's archive is small (the job history parquet files total under 200 KB), so these reads never came near any limit; the numbers show thread scaling, not out-of-memory behavior.

Tests (Lite.Tests, targeted): DuckDbMemoryLimitSettingTests 53 total, 0 failed (default, range, fallback table, LoadAtStartup, connection string and a real connection carry the setting, COPY raise below and above 4 GB and the restore, trim restore, thread formula, threads after trim and COPY raise, preserve_insertion_order, out-of-memory message names the setting and value including inner exceptions, other errors untouched, save report objection, Settings window source pin). Also run: MainConnectionLimitsTests 4/0, *Archive* 255/0, *DuckDb* 99/0, *Census* 58/0, *Settings* 82/0, *Compaction* 75/0 (before the thread change), -trait Stage=Guard -failSkips 429/0. No *Ratchet* class exists in Lite.Tests (0 matched). 0 build warnings. I did not run the full suite.

Not done: the Settings window box and the dialogs are checked by source pins and the build, not by running the WPF window. The out-of-memory sentence is added at the DuckDB-reading surfaces listed above, not at every one of the ~85 catch blocks in Lite's controls that only log; the log line of those is are covered by the AppLogger change above.

CHANGELOG (this part)
SECTION: Added
ENTRY: - Lite has a DuckDB memory limit setting ([#5553]) - Settings, Dashboard Defaults, "DuckDB memory limit" sets the main DuckDB connection's memory limit (default 2 GB, range 1 GB to 80% of memory, applied after a restart); the thread count follows it as the smaller of 8, the core count and the limit divided by 256 MB, and out-of-memory errors name the setting and its current value.

… not after it (#5457) v_job_history dedups the hot table and the archive on (server_id, instance_id). DuckDB moves a filter below that window only when the filter reads PARTITION BY columns, so each Job History read sorted a server's whole archive, message text included, before its run_datetime filter could drop a row. run_datetime joins the key: every copy of a run carries the same value, so the same copies collapse, and the filter now reaches the table and parquet scans.

…d a run archived and live still collapses (#5457)

…iewer calls that awaited it on the dispatcher move into Task.Run, and a census holds Lite/Controls to it (#5457)

…census covers the main window (#5457)

… memory, applied at startup (#5457) The connection string, the COPY raise and restore and the trim restore follow the setting, and a DuckDB out-of-memory error shown in a tab names the setting and its current value.

…ix/5457-mem-setting

… the Database State Overrides editor, the history and drill-down windows, and the census holds Lite/Windows to it (#5457)

…ix/5457-windows-offui

…res, limit / 256 MB) (#5457)

…ix/5457-mem-setting

…ory line (#5457) AppLogger appends the memory-limit hint once to a WARN or ERROR line that holds DuckDB's exact out-of-memory text, so the catch blocks that only log the raw message name the setting too.

Adds the changelog entries for the 3.10.0 PRs that merged after the first splice: #5528, #5532, #5537, #5542, #5543, #5552 and #5553. Each full entry goes into docs/changelog/3.10.md and its title-only line into the CHANGELOG.md index. The archive census row for 3.10.0 moves from 257 to 265 entries. Changelog entries for the 3.10.0 PRs merged after the first splice.

…kDB memory limit Release checklist step 2b. README.md already covered everything up to #5491. This adds the three user-facing 3.10.0 changes merged after it: the one time range picker (#5570), the Excel-style text column filters (#5577) and Lite's DuckDB memory limit setting (#5553). Collector count, tab lists, edition table, platform notes, permissions and the deprecation callout needed no change.

erikdarlingdata added 4 commits October 8, 2026 06:23

Pin the Job History dedup: the run time filter reaches both scans, an…

0b5554b

…d a run archived and live still collapses (#5457)

Lite tabs read the store off the UI thread: the AG, FinOps and plan-v…

b5aa915

…iewer calls that awaited it on the dispatcher move into Task.Run, and a census holds Lite/Controls to it (#5457)

The Overview refresh reads the fleet tags off the UI thread, and the …

9af7479

…census covers the main window (#5457)

erikdarlingdata changed the title ~~Lite's Job History reads filter on run time before the archive dedup (#5457)~~ Lite's Job History reads only the selected time range, and tab loads read off the UI thread (#5457) Oct 8, 2026

erikdarlingdata marked this pull request as ready for review October 8, 2026 10:48

erikdarlingdata added 7 commits October 8, 2026 06:57

Merge remote-tracking branch 'origin/fix/5457-lite-duckdb-oom' into f…

62e7845

…ix/5457-mem-setting

Lite windows read and write the store off the UI thread: Manage Tags,…

f7a25c2

… the Database State Overrides editor, the history and drill-down windows, and the census holds Lite/Windows to it (#5457)

Merge remote-tracking branch 'origin/fix/5457-lite-duckdb-oom' into f…

f330290

…ix/5457-windows-offui

Lite derives DuckDB's thread count from the memory setting: min(8, co…

f305bd9

…res, limit / 256 MB) (#5457)

Merge remote-tracking branch 'origin/fix/5457-lite-duckdb-oom' into f…

80a679f

…ix/5457-mem-setting

Lite's log names the DuckDB memory setting on every DuckDB out-of-mem…

ac4e021

…ory line (#5457) AppLogger appends the memory-limit hint once to a WARN or ERROR line that holds DuckDB's exact out-of-memory text, so the catch blocks that only log the raw message name the setting too.

erikdarlingdata merged commit d78f4a4 into dev Oct 8, 2026
24 of 26 checks passed

erikdarlingdata deleted the fix/5457-lite-duckdb-oom branch October 8, 2026 11:50

erikdarlingdata mentioned this pull request Oct 8, 2026

Release 3.10.0: changelog entries for the last fixes #5557

Merged

8 of 9 tasks

erikdarlingdata mentioned this pull request Oct 9, 2026

Release 3.10.0: last changelog splice and README sync #5635

Merged

Navigation Menu

Repository navigation

Lite's Job History reads only the selected time range, and tab loads read off the UI thread (#5457) - #5553

Lite's Job History reads only the selected time range, and tab loads read off the UI thread (#5457)#5553
erikdarlingdata merged 11 commits into
devfrom
fix/5457-lite-duckdb-oom

erikdarlingdata commented Oct 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

erikdarlingdata commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Measured (synthetic store, scratch folder only, DuckDB 1.5.5)

What the first tab switch waits on

Test plan

Not done

CHANGELOG

DuckDB memory setting

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

erikdarlingdata commented Oct 8, 2026 •

edited

Loading