Column filters list a column's values to tick, like Excel, and stay set across refreshes and restarts

[erikdarlingdata]

A text column's filter now lists the values in that column, like the filter in Excel. You tick the values you want to see, untick the ones you do not, search the list, and use Select All. Ticking several logins gives the IN() list the reporter asked for, and the text match stays beside the list. The filter stays set when a grid refreshes, when you switch tabs, and when you close and reopen the app. The web pages part is below; the desktop part is added next to it.

SECTION: Added
ENTRY: - Column filters list a text column's values to tick, like Excel, and stay set across refreshes and restarts ([#5577]) - A text column's filter lists its values, with a search box, Select All and "(Blanks)", in Lite, the Darling viewer and the web dashboard. Unticking values hides only them, so a new login after a refresh still shows, and ticking values shows only those. The filter stays set across refreshes, tab switches and restarts, per server, grid and column. Lite and the viewer keep it in a file next to their settings, and the web dashboard keeps it in the browser. Number and time columns keep the text match only. Query text, plan, XML, message, detail, description, reason, error and script columns also keep the text match only, and that match lasts for the session.

Web pages

What changes in the web grids (Darling/PerformanceMonitor.Darling.Service/wwwroot/):

• js/grid-value-filter.js (new): the value part of a filter as { mode, set, blank } (mode none, hide or showOnly), the distinct-value list (ordinal, ignoring case, first spelling stands, sorted), the match, the mode choice when ticks change (all ticked is none; fewer unticked than ticked hides the unticked; otherwise shows only the ticked), the chip words ("hides 3 values", "shows 2 values"), and the kept copy: serializeFilters and parseFilters.
• js/panels.js (gridFilter): the popup gets a search box, a Select All checkbox, "(Blanks)" first when any cell is null, empty or whitespace, one checkbox per value, a note "Showing 1,000 of N values. Search to find the rest." above 1,000 values, and "No values are ticked, so no rows show." A row has to pass the text match AND the value part, and columns still AND together. The column's button shows active when either part is. Every value is a text node (el(..., { text })), the checkboxes carry aria-labels, Escape closes the box, and the 60 s repaint and an in-place reconcile (Alert History) both re-list an open box. Clear (in the box), the chip's x and Clear all filters clear both parts.
• The list is offered on text columns only, and only when no value is over 256 characters. Query, statement and SQL text, XML and index-definition columns carry valueList: false (about 20 descriptors in js/pages/server-tabs.js, pages/finops/high-impact.js, index-analysis.js, optimization.js and js/read-fields.js); plan columns already had filter: false.
• The kept copy is localStorage key pm.gridFilters.v1: the active filters only, per route (which carries the server) + grid + columns, at most 200 grids (the least recently used go first). Every read and write is in try/catch; a missing, unreadable or refused store starts empty, never stops a grid from drawing, and warns once. A stored shape is checked field by field on the way in.
• css/app.css: .grid-filter-values, .gfv-*, using the theme's variables only.
• A refresh and a tab switch already kept the filters in the page (module state keyed by route + grid + columns); the new tests pin that they still do.

Decisions worth a second look:

• Select All ticks or unticks every value the search matches, not only the first 1,000 listed. With no search it covers everything, including "(Blanks)". The desktop part may want to match this.
• A column whose value part holds more than 1,000 values is filtered in the page but its value part is not kept across a restart (cutting a set would hide the wrong rows). The text match of that column is still kept.

Desktop (Lite and the Darling Viewer)

Lite and the Viewer share the filter code (PerformanceMonitor.Common, PerformanceMonitor.Ui), so every filtered grid in both gets the list, and both apps are wired the same way.

• PerformanceMonitor.Common/Models/ColumnFilterState.cs: the state gains a value part, ValueMode (None, ShowOnly, Hide), Values (ordinal, ignoring case) and ValueBlank (a flag of its own, so a real value "(Blanks)" stays a normal value). IsActive is the text match or the value part; the chip text reads "hides 3 values", "shows 2 values" plus the text match.
• PerformanceMonitor.Common/Services/ColumnFilterMatcher.cs: a row passes when it passes the text match AND the value part (null, empty and whitespace cells are the blank flag's).
• PerformanceMonitor.Common/Services/ColumnValueList.cs (new): ColumnValueCatalog (distinct, ignore case, sorted, blank flag, 1,000 cap and its note, search, the 256-character limit) and ColumnValueSelection (the mode choice, and a stored value the rows no longer hold is kept, so an unchanged Apply does not drop it). ColumnValueListColumns.cs (new) leaves query, statement, SQL, plan, XML, prose and display-string columns (names ending Text, Formatted, Display, Message, Definition, and so on) to the text match.
• PerformanceMonitor.Ui/ColumnFilterPopup.xaml(.cs), ColumnValueListModel.cs (new): a search box, Select All (three-state), a virtualised checklist, the cap and no-ticks notes, "(Blanks)" first, the text match kept below it, and "Clear all filters". Tab order follows the layout, the list is one tab stop with arrow keys inside, Space ticks, Esc closes from anywhere, Enter applies. It uses the same theme brushes the old popup did, so Lite's and the Viewer's themes both apply.
• DataGridFilterManager.cs: values come from the rows the grid holds before any filter; no list for a non-string property, a column that sorts by another member (a time shown as text), a value over 256 characters, or an excluded name. UpdateData already re-applied filters; it now also loads the stored ones on the first refresh under a server scope. ClearFilters (the Viewer grid column filters survive Refresh and server switches — a stale filter silently empties a grid while its counts stay full #2306 server-switch path) leaves the stored copy alone, and the next refresh loads the new server's; the new ClearAllFilters (the user's button) also removes the stored copy.
• ColumnFilterStore.cs, ColumnFilterScope.cs (new): one JSON file per app (column-filters.json beside Lite's settings.json, and beside the Viewer's viewer-preferences.json), written 1.5 seconds after the last change, by temp file then replace, and flushed on exit. Key: server + grid + column. Active filters only. At most 200 grids (least recently used go first); a column naming more than 1,000 values is not stored at all (cutting a set would hide the wrong rows), though it still filters in the grid. A file that cannot be read is logged once and ignored; a store that throws never blocks a grid. The file holds the ticked or typed values and the server, grid and column names, nothing else.
• Server scope: an inherited attached property set on each server tab's root (ServerTab: the server's id; ViewerServerTab: the server name) and the grid's Name, so the 105 filter managers needed no one-by-one edit. The FinOps tab in each app has its own server picker, so its scope follows the picker (set before the same-server early return). ColumnFilterStore.Install is called once in each app's startup.
• Not stored, kept for the session only (no server scope): the Alert History and Job History tabs and the drill-down windows (Wait Drill Down, Query / Procedure / Query Store history). Their filters still survive a refresh.

Checked in both apps: the grid names (a census reads every DataGridFilterManager under a scope and fails on a grid with no x:Name in its XAML or two grids with one name under one scope; none today), both startup installs, both scope sets, the FinOps pickers, and that neither app carries its own filter class.

Test plan

• GridValueFilterBehaviourTests (new): Total 19, Failed 0. It drives all 10 shared cases in Fixtures/column-value-filter-cases.json through the real boxes in Node (tick, untick, tick only, Select All, search, text match, refresh), plus Select All under a search, the 1,000 cap note, a value like <b>x</b> built as text, which columns get a list, labels and Escape, chips, a reconcile in place, the kept copy (round trip, refresh, tab switch, restart, Clear all), a bad or refused store, and the bounds (200 grids, 1,000 values).
• RED: with panels.js reverted to dev, all 19 fail; restored, all 19 pass.
• GridFilterBehaviourTests (the existing text filter): Total 15, Failed 0. WebModuleLoadTests Total 7, Failed 0 (it caught a comment naming an export; reworded). The *Web* classes ran Total 1,619 with that one failure (Skipped 62); after the fix WebModuleLoadTests was re-run alone, and the new and filter classes again after merging dev.
• *Census* Total 477 Failed 0; *Ratchet* Total 15 Failed 0; *Hygiene* Total 90 Failed 0; *Scrub* Total 120 Failed 0; *Guard* Total 338 Failed 0 (Darling.Tests, one class at a time).
• Darling.Tests builds with 0 warnings.
• Not run: the full suite (CI decides), and a look at the page in a browser (no app was run).

Desktop

• ColumnValueFilterCasesTests (new): Total 11, Failed 0. It drives all 10 cases of Fixtures/column-value-filter-cases.json through the shared code (the web part reads the same file).
• ColumnValueListTests (new): Total 35, Failed 0 (list builder, cap and search notes, mode choice, the match, chip text, excluded columns, Select All under a search past the cap).
• ColumnFilterPersistenceTests (new): Total 25, Failed 0 (store round trip, per-server separation, least-recently-used bound, 1,000-value bound, bad file ignored and logged once, a cleared filter removed, debounce and atomic replace; the grid manager across UpdateData, a restart, a server switch, "Clear all filters", and a store that fails).
• ColumnFilterPopupTests (new): Total 8, Failed 0 (the real popup controls). ColumnFilterWiringCensusTests (new): Total 6, Failed 0.
• RED: with the value part left out of ColumnFilterMatcher, 12 of the fixture and list tests fail; with the stored-filter load switched off, 4 of the persistence tests fail. Both restored.
• Darling.Tests *ColumnFilter*, *ColumnValue*, *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* on the merged head: Total 1,050, Failed 0, Skipped 60. Lite.Tests the same five wildcards plus the grid and server-tab pins: Total 266, Failed 0. Lite, the Viewer and both test projects build with 0 warnings.
• Not run: the full suites (CI decides), and the apps (no window was opened, so the popup's look in each theme is unchecked beyond reusing the old popup's brushes).

Round 2

Head d40edec. The desktop half is done and tested; the web page half is partly done (see Not done).

Done on the desktop (Lite and the Darling Viewer)

• The mode (None, Hide, ShowOnly) is chosen from the ticks on the listed entries only, as the web page does. A value the stored filter names that the rows no longer hold no longer tips the count, and it is carried over only when the new mode is the stored filter's own. (ColumnValueList.cs, ApplyTo.) New shared fixture case 11 runs it on both halves.
• Apply rebuilds the value part only when a tick changed (ColumnValueListModel.TicksDirty); a text-only Apply keeps the stored value part. Select All and Tick only now raise TicksChanged, so the "No values are ticked" note shows at once.
• A stored filter naming over 1,000 values keeps its text match and loses only the value part, as the web page does.
• The store applies every limit when it reads the file too (file size, 200 grids, 200 columns per grid, 1,000 values, value and text length, key length, undefined operator or mode). A null or wrong-typed entry is skipped on its own; a file that cannot be read safely is logged once and ignored whole.
• A failed save stays dirty and is retried on the next debounce tick; the warning is given once per run. The old "replace" test is now an atomic-write test: with the temp file blocked, a plain write would change the old file, the replace path leaves it byte for byte the same.
• A text match on a column that gets no list (name rule: query, statement, plan, XML, message, detail, description and the rest) is kept for the session only and is never written or read back. The shared name rule gained the Script, Error and Info suffixes.
• Alert History and Job History, in Lite and in the Viewer, share one fixed all-servers scope (ColumnFilterScope.AllServers) so their filters survive a restart. The drill-down windows get no scope.
• A stored value part on a column that no longer gets a list shows in the popup as a read-only line with "Clear the value filter".
• The Viewer's column-filters.json now lives under Local AppData; a file in the old Roaming folder is moved over once and then gone.
• Shared fixture: cases 11 (mode with gone values), 12 and 13 (case folding like .NET OrdinalIgnoreCase: the sharp s is not "ss"; the dotless i and long s stay themselves), the mode rule text, and a list of desktop/web column-name pairs that both halves must answer the same way.

Done on the web page

• grid-value-filter.js: valueKey folds like OrdinalIgnoreCase (fixture cases 12 and 13 pass through the real boxes); the no-list name rule (NO_LIST_SUFFIXES, NO_LIST_FRAGMENTS, noListName, listableColumn) exists; serializeFilters skips a column flagged session and any column named like prose; parseFilters drops such columns. The test harness now finds a value by its exact spelling first.
• GridValueFilterBehaviourTests: a missing Node fails on CI (GITHUB_ACTIONS is set) instead of skipping. In a recent dev run's log (run 37814992459) no page-test class was skipped (the skips there are PostgreSQL live tests), so this only guards the future.

Test plan

• *ColumnFilter* Total 52, Failed 0; *ColumnValue* Total 51, Failed 0; *GridValueFilterBehaviour* Total 22, Failed 0.
• *Census* 483/0 (1 skipped), *Hygiene* 90/0, *Ratchet* 15/0, *Scrub* 123/0 (32 skipped), *Guard* 338/0 (27 skipped).
• Seen failing without the fix (production lines reverted, then restored): the mode count, the dirty-flag Apply, the no-list text exclusion, the over-cap value part, the save retry, and the Select All note each fail the new test that names them: 10 tests failed in the reverted build.
• Lite.Tests grid and server-tab classes were not run. (run in round 2b: see below)
• WebModuleLoadTests was not run. (run in round 2b: see below)

Not done (the web page half and one test)

• panels.js is unchanged: not yet done are (a) using listableColumn in listed() and the opt-out valueList: false plus the name rule, (b) raw-value cells for the list (blank means a null or empty raw value, whatever the cell renders; documented equivalence for plain text columns), (c) the search folded with valueKey, (d) writes to localStorage debounced about 300 ms and flushed on pagehide (the harness stored() then needs flushGridFilters), (e) the session flag on a text match for a no-list column, plus pruning loaded entries for a valueList: false column, (f) the read-only "kept value filter" line with a Clear button in the web popup, (g) FinOps: rewrite the URL to #/finops/{server}/{tab} with history.replaceState in paint() so the filter key carries the server, with a test. (done in round 2b)
• A census test comparing NO_LIST_SUFFIXES/NO_LIST_FRAGMENTS to ColumnValueListColumns.NameSuffixes/NameFragments, and a fixture-pairs test on both halves (the C# properties exist; the tests are not written). (done in round 2b)
• A test that Alert and Job History tabs set the all-servers scope in both apps and that the three HistoryDataGrid windows set none (the persistence behaviour is tested; the pin is not). (done in round 2b)
• The valueList: false marker on the reason column (server-tabs.js:3123). (done in round 2b)
• The squash note was not changed beyond the earlier text; the CHANGELOG ENTRY stands. (done in round 2b)

Round 2b

The web page half of round 2 is finished and the "Not done" list above is ticked off. All of it is in panels.js and grid-value-filter.js unless a file is named.

Done on the web page

• No-list name rule in the list decision. gridFilter now decides with listableColumn (the valueList: false marker plus the shared name rule), so the Alerts Detail column (detail_text), job step and severe-error messages, recommendation details, descriptions, scripts, errors and info cells get no list even when every cell is short. (panels.js, listed.)
• Raw values. The list is built from each cell's raw value, not the text the cell draws; a cell is blank when its raw value is null, empty or whitespace. For a plain text column the two are the same string, so nothing changes there; they differ only where a column draws something other than its value, and then the raw value is what the desktop list reads. A list or object value has no text of its own, so the drawn text stands for it. (listCell.)
• Search. The value search folds case with valueKey, like .NET OrdinalIgnoreCase: a capital sharp s no longer matches a lower-case one.
• Debounced writes. The kept copy is written after 300 ms of quiet and flushed on pagehide (flushGridFilters is exported for the tests).
• Session-only text. A text match on a column that gets no list is marked session: it stays through refreshes and tab switches, is never written, and a stored one on such a column is dropped when the grid is drawn. serializeFilters keeps only the value part of such a column if it still has one.
• Kept value part on a column with no list. The filter box shows "A kept value filter hides N values. This column has no value list." with a "Clear the value filter" button. The value part still filters until cleared.
• FinOps. pages/finops.js rewrites a bare #/finops to #/finops/{server}/{tab} with history.replaceState (pinFinopsHash), before the tab builds, so each server has its own filter key.
• reason columns. The stored-caveat reason column (server-tabs.js) and the mute rule reason column (mute-rules.js, free text typed by a user) are marked valueList: false. A key ending in "reason" is not in the shared name rule, which the desktop pins to the same list.
• Small fixes in the same files: the open-list refresh test now changes the row object as the real reconciler does; a style for the kept-value line in app.css.

New tests (each seen failing with its production line reverted)

Test	Reverted line	Result
ColumnValueListNameRuleTests (suffix and fragment lists equal on both halves; desktop answers for the fixture's name pairs)	removed "info" from NO_LIST_SUFFIXES	the suffix test fails
GridValueFilterBehaviourTests.TheNoListNameRule_...EveryPairInTheSharedFixture (web half of the pairs)	same	fails
...AColumnNamedLikeProse_OrAStatement_GetsNoList_EvenWithShortCells	listed back to valueList !== false	fails
...TheList_HoldsTheRawValue_...	list built from textContent again	fails
...TheValueSearch_FoldsCaseLikeDotNetOrdinalIgnoreCase	search back to toLowerCase	fails
...TheKeptCopy_IsWrittenOnceAfterABurst_AndAtOnceOnPagehide	write on every change; no pagehide listener (two reverts)	fails both times
...ATextMatch_OnAColumnWithNoList_LastsTheSessionOnly...	removed merged.session = true	fails
...AValuePartKeptOnAColumnThatHasNoList_...	no kept line; no prune of stored text (two reverts)	fails both times
...OnABareFinOpsAddress_EachServerGetsItsOwnFilters	replaceState call removed	fails
...TheFinOpsPaint_PinsTheServerBeforeItBuildsTheTab (source pin: the page cannot run without a server)	pinFinopsHash call removed	fails
...TheFreeTextReasonColumns_AreMarkedWithNoList (source pin: those descriptors sit in page modules that need a server)	marker removed	fails
ColumnFilterWiringCensusTests.Alert_History_and_Job_History_set_the_one_all_servers_scope_in_both_apps	removed the scope line in Lite Alert History, then in the Viewer Job History	fails both times
...The_three_HistoryDataGrid_windows_in_each_app_have_no_scope	added a scope to one window	fails

The earlier round's atomic-write test (A_write_goes_through_a_temp_file_so_a_failed_write_leaves_the_old_file_whole) was also checked: with ColumnFilterStore writing straight into the real file it fails (and so does the retry test), so it does test the temp-file-and-replace path.

Test plan

• *ColumnFilter* Total 54, Failed 0; *ColumnValue* Total 55, Failed 0; *GridValueFilterBehaviour* Total 32, Failed 0; *GridFilter* Total 15, Failed 0; WebModuleLoadTests Total 7, Failed 0 (it caught a trailing comment with the word "tab" in finops.js; moved onto its own line).
• Darling.Tests: *Census* 485/0 (1 skipped), *Ratchet* 15/0, *Hygiene* 90/0, *Scrub* 123/0 (32 skipped), *Guard* 338/0 (27 skipped).
• Lite.Tests: *Grid* 168/0, *ServerTab* 84/0, *Census* 63/0, *Scrub* 32/0, *Guard* 136/0 (no ColumnFilter, Ratchet or Hygiene classes there).
• Both test projects build with 0 warnings.
• The full suites were not run on this machine; CI decides.

Notes

• The text match on a number or time column is still kept across restarts: only columns that get no list by name or by the valueList: false marker are session-only, as on the desktop.
• A stored filter on a column that the name rule now covers (for example one ending in "Script") is dropped whole when read, text and values, as on the desktop.

Round 3

Six fixes from the second check. Each behaviour has a test that fails with its production line reverted. The round 2b line about the two reason columns carrying valueList: false is replaced by item 4 below.

1. Columns built from other fields list what their cells show (panels.js, listCell). A row with no raw field for the column (undefined), under a column with render or display, uses the shown text, as before the list read raw values. The Alerts Status, Triage and Mute columns list their labels again (Status: Delivered, Delivery error, Muted); unticking "Muted" hides only the Muted rows, and a stored "Muted" filter matches them after a reload. A valueList: false column still gets no list. Test: AColumnBuiltFromOtherFields_ListsTheTextItShows_AsTheAlertsStatusColumnDoes (harness scenario derivedColumn). Reverted: the raw === undefined branch; the harness cannot find the "Muted" item and fails.
2. Desktop "Clear the value filter" applies at once (ColumnFilterPopup.xaml.cs): the button now calls ApplyFilter(), as the web does, so a click outside or Escape keeps the cleared state. Test: A_stored_value_part_on_a_column_with_no_list_shows_read_only_with_a_way_to_clear_it now checks that the event fires on the click. Reverted: the ApplyFilter() call.
3. Mute rules form field list (mute-rules.js): removed the valueList: false on the form's reason field (the rules table never went through a grid), and removed its source-text pin.
4. "reason" in the shared no-list suffix list on both halves (ColumnValueListColumns.cs, grid-value-filter.js). The explicit marker at server-tabs.js (stored caveats) is gone: the name rule covers it. The fixture gained Reason and RecommendationReason pairs, so both halves are tested; the census tests keep the lists equal. The offeredByName scenario no longer sets a marker on reason. Reverted: the C# suffix (suffix-list and pair tests fail) and the JS suffix (AColumnNamedLikeProse_... fails, reason offered).
5. Desktop store retry (ColumnFilterStore.cs): a failed save retries after 1.5 s, then 3, 6, 12, 24, 48 and at most 60 s (a good save resets it; RetryDelay shows the next wait). The file I/O now runs outside the state lock, under its own write lock, so Load and Save never wait on the disk; a change made during a write keeps the store dirty. The warning now says what happens: "The save is tried again in N seconds, with longer waits (up to 60 seconds) while it keeps failing, and at the next change." Tests: A_save_that_keeps_failing_backs_off_... (10 s debounce: 10, 20, 40, 60, 60, 60 then reset) and A_slow_write_does_not_hold_up_Load_or_Save (a blocked writer; Load and Save finish). Reverted: the doubling; the write moved back under the lock (the second test fails).
6. Duplicates on read (ColumnFilterStore.cs): the last copy of a server and grid in the file wins, and a column named twice in a grid keeps its last copy, so Load and Save see one entry. Test: The_same_server_and_grid_twice_in_a_file_keeps_the_last_copy_so_Load_and_Save_agree (a hand-built file). Reverted: the entry-level removal (the column-level dedupe was not reverted on its own; the same test asserts it).

The squash text stays true: it lists no reason column and says a kept value filter shows a Clear button, which still holds.

Test plan (round 3)

• Darling.Tests: *ColumnFilter* 57/0, *ColumnValue* 55/0, *GridValueFilterBehaviour* 32/0, *GridFilter* 15/0, WebModuleLoad* 7/0, *Census* 485/0 (1 skipped), *Ratchet* 15/0, *Hygiene* 90/0, *Scrub* 123/0 (32 skipped), *Guard* 338/0 (27 skipped).
• Lite.Tests: *Grid* 168/0, *ServerTab* 84/0, *Census* 63/0, *Scrub* 32/0, *Guard* 136/0 (no ColumnFilter, Ratchet or Hygiene classes there).
• Both test projects build with 0 warnings.
• The full suites were not run on this machine; CI decides.

Fixes #5565