Skip to content

[BUG] PMLite 3.5.0: Refresh, Deadlocks, Database sizes #2640

Closed
Closed
[BUG] PMLite 3.5.0: Refresh, Deadlocks, Database sizes#2640
Labels
bugSomething isn't working

Description

@ZedZipDev
ZedZipDev
opened on Aug 26, 2026

Component

Lite

Performance Monitor Version

3.5.0

SQL Server Version

Azure SQL DB

Windows Version

W11

Describe the Bug

  1. Last 7 days or other -> after restarting PMLite-> Last 4 hours

  2. FinOps-> Database sizes: shows only 2 master database files:
    master data_0
    master log

  3. Blocking->Deadlocks
    I see 4 deadlocks:
    2 : 2026-08-20
    2 : 2026-08-21

But really query returns me deadlocks for 22, 23, 24, 25, 26 Aug.

Steps to Reproduce

Run PMLite, exit PMLite, run PMLite

Expected Behavior

The Last xx restored as it was before exit

Database Sizes: all N databases in Azure SQL DB, on-premise SQL Server, AWS RDS
and probably including tempdb

Actual Behavior

As described

Error Messages / Log Output

Screenshots

No response

Additional Context

No response

Activity

  1. ZedZipDev
    added
    bugSomething isn't working
    on Aug 26, 2026
  2. ZedZipDev
    changed the title [-][BUG] PMLite 3.5.0[/-] [+][BUG] PMLite 3.5.0: Refresh, Deadlocks, Database sizes[/+] on Aug 26, 2026

  3. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    Also, the Database sizes

    Image

  4. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026
    Owner

    Thank you — three separate defects, and all three are real. Two are fixed and merged (#2642); the third is a genuine durability gap and has its own issue (#2641).

    1. Time range not restored — fixed

    The picker wrote nowhere. default_time_range_hours already existed and was already read at startup, but only the Settings window ever wrote it, so choosing "Last 7 days" and restarting came back at four hours with nothing to say why. Picking a range now persists it, and a second server tab opened in the same session uses it too.

    Custom Range is deliberately not persisted — it has no hours value to store, and restoring a window that ended two days ago would open the app on an empty chart of a range you had moved on from. If you want that, say so and I will store the absolute window instead.

    2. Database Sizes showing only master's two files — explained, and the display fixed

    This one is working as designed, and the design was invisible. On Azure SQL DB the collector reads sys.database_files on the connected database only. That is deliberate: it used to enumerate databases first, and that enumeration connects to master, which is the one database an Azure login reaching the server through a database-level firewall rule cannot open (#1631). On Azure SQL DB a login can only see its own database anyway, so the enumeration bought nothing and cost a failure.

    You were connected to master, so you got master's two files — correct, and indistinguishable from a collector that only found master.

    The grid now says so rather than leaving you to work it out:

    2 file(s) — Azure SQL Database: only the CONNECTED database ('master') is visible from this connection, so its siblings are not missing, they are unreachable. Add a server entry per database to see more.

    To actually see your databases: add a server entry per database. There is no connection on Azure SQL DB that can see them all, so "all N databases" is not reachable from one entry — that is the platform, not the tool. (On on-prem and AWS RDS one entry does cover every database, including tempdb, via the sys.master_files path.)

    3. Deadlocks stopping at Aug 21 — filed as #2641

    The most substantive of the three. On Azure SQL DB our only deadlock source is a database-scoped XE session we create, read from its ring buffer, with max_memory = 4096 (4 MB) and ALLOW_SINGLE_EVENT_LOSS. A ring buffer is memory-resident, so any Azure failover or scale operation empties it — Azure does those unannounced — and Lite collects only while it is running. Between those, a deadlock that happened while the app was closed is gone.

    Meanwhile Azure's own telemetry_xevents session is file-backed and retains far more, and this product has never read it — git grep for telemetry_xevents or fn_xe_telemetry_blob_target_read_file returns nothing. That is almost certainly where your Aug 22–26 deadlocks are, and it is the Azure analogue of system_health, which we already parse on every other platform.

    One thing that would confirm it: was Lite running during Aug 22–26, or closed? That distinguishes "the ring buffer was cleared by a failover" from "nothing was collecting" — both are covered by reading the durable source, but it tells me which one you actually hit, and whether the buffer size is worth raising as well.

  5. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026 via email
    Owner

  6. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    Yes, and yet one: when I go to Database Resources I wee all 50 databases in the Azure SQL DB.
    You know better but may be go db by db in the Database sizes and get sizes? Or may be get db sizes but not file sizes? As for me it wil be enought to see the db size.

  7. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    One issue per issue please.

    /*
    I typed this on a phone.
    Thanks for understanding.
    */
    …

    Yes, sir, my laziness.....

  8. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    About 3. Yes, PMLite worked all these days 24, 25, 26 Aug 24 hours /day. Btw, Webhook automatically restored the broken connection N times per day- it works OK.

  9. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026
    Owner

    Correcting myself on the database sizes — I was wrong, and your suggestion is the right one.

    I said "there is no connection on Azure SQL DB that can see them all — that is the platform, not the tool." That is not true, and I checked it against a live Azure SQL Database instead of reasoning about it a second time. Connected to master:

    sys.databases       | every database on the logical server
sys.resource_stats  | per database, WITH SIZE

database_name | storage_in_megabytes | end_time
Erik          | 19090                | 2026-08-26 12:21:09

    sys.resource_stats is a master-only view carrying storage_in_megabytes per database with roughly 14 days of history. So exactly what you asked for — "maybe get db sizes but not file sizes? As for me it will be enough to see the db size" — is available, and it is the same reason Database Resources shows you all 50.

    Filed as #2643 with the measurements. What stays true is the narrower point: per-FILE detail for a database you are not connected to is not reachable, because sys.database_files is database-scoped. Size is; files are not.

    Thank you for pushing back on it. The tab explaining its scope (#2642) was the wrong fix on its own — an explanation for a limit that is not real.

    On the deadlocks, your "PMLite ran 24 hours/day on 24, 25, 26" rules out both of my theories, which is exactly why I asked. It was not "the app was closed", and a 4 MB ring buffer would not silently discard three days on a quiet database while keeping the oldest four events. Something is stopping collection while the app is running, which is a different and more serious bug than the one I filed. I have retitled my thinking on #2641 accordingly.

    Two questions, and they are the last ones I need:

    1. Which database do the deadlocks occur in, and which database is the PMLite server entry connected to? On Azure SQL DB our deadlock session is database-scoped — a session created in master captures only master's deadlocks. If your entry points at master and the deadlocks are in a user database, that alone explains all of it, and it is our bug rather than your configuration.

    2. What query returns the Aug 22–26 deadlocks for you? I tested sys.fn_xe_telemetry_blob_target_read_file on a real Azure SQL Database today — every prefix (dl, deadlock, xe, errors, and six others) returns zero rows, so that is not the source, and knowing what you actually ran tells me where they really live.

    I created a genuine deadlock on a test Azure SQL Database while checking this; our ring buffer captured it correctly, so the read path itself works. The failure is somewhere between that and your store, and your answer to (1) is most likely to name it.

  10. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    1 & 2
    The query

    WITH CTE AS (
SELECT CAST(event_data AS XML)  AS [target_data_XML]
FROM sys.fn_xe_telemetry_blob_target_read_file('dl', null, null, null)
)
SELECT target_data_XML.value('(/event/@timestamp)[1]', 'DateTime2') AS Timestamp,
target_data_XML.query('/event/data[@name=''xml_report'']/value/deadlock') AS deadlock_xml,
target_data_XML.query('/event/data[@name=''database_name'']/value').value('(/value)[1]', 'nvarchar(100)') AS db_name
FROM CTE
order by 1 desc--db_name
GO

    it returns

    Image

    And check PMLite shows only 2 deadlocks

    Image

  11. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026
    Owner

    Your query caught me out, and thank you for posting it — I was wrong twice in a row on this one.

    sys.fn_xe_telemetry_blob_target_read_file is master-scoped. My test connected to a user database, got zero rows for every prefix, and I reported the source as unusable. Same server, same 'dl' prefix, same minute:

    connected to master | 1 row
connected to a user database | 0 rows

    Running your query from master against a deadlock I induced returns everything a collector needs — the event's timestamp, the user database's name, the victim process id, the full 15 KB graph, and a blob .xel file name. File-backed and server-wide, which our own session is neither.

    That almost certainly explains your whole report. Our Azure deadlock capture uses a database-scoped Extended Events session — a session created in master sees only master's deadlocks. Your entry is connected to master, your deadlocks are in the fifty user databases, so we captured essentially none of them while the durable source that sees all fifty sat unread. The four from Aug 20–21 are the exception rather than the rule.

    Two answers I no longer need to ask you for: which database, and what query. You supplied both.

    Tracking on #2641 with the verified output. The fix is to read that function from a master connection alongside the ring buffer, keyed on (deadlock_time, victim_process_id) so the two sources merge rather than duplicate.

    Summary of where your three reports stand:
    1. Time range not restored Fixed and merged (#2642)
    2. Database sizes Display scope fixed (#2642); your actual ask — per-database sizes from master via sys.resource_stats — filed as #2643, and you were right that it is possible
    3. Deadlocks Root cause identified on #2641: database-scoped session vs. a master-scoped durable source

    All three were real, and two of them I would have got wrong without your pushback.

  12. ZedZipDev commented on Aug 26, 2026

    @ZedZipDev
    ZedZipDev
    on Aug 26, 2026
    Author

    Yes, sorry, I forgot to add: I run the query in master, but you already wrote.

  13. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026
    Owner

    Both follow-ups from this report are now shipped:

    Both verified against a live Azure SQL Database rather than reasoned about. Thank you for the pushback on all three — two of them I would have closed wrongly without it.

  14. erikdarlingdata commented on Aug 26, 2026

    @erikdarlingdata
    erikdarlingdata
    on Aug 26, 2026
    Owner

    All three are shipped:
    1. Time range not restored #2642
    2. Database sizes #2642 (scope explained) + #2648 (every database's size from master — what you actually asked for)
    3. Deadlocks #2644 (Azure's durable telemetry, read from master, carrying the user database's name)

    Every one verified against a live Azure SQL Database rather than reasoned about. Two of them I had answered wrongly first, and your pushback — the sys.resource_stats observation and posting your deadlock query — is what corrected both.

    Closing this one; #2645 (row marks) is also shipped, and #2647 is Lutz's, separate.

  15. erikdarlingdata
    closed this as completedon Aug 26, 2026
  16. erikdarlingdata
    added a commit that references this issue on Sep 16, 2026

    Remember the auto-refresh toggle and interval, the way the time range…

    4c528a3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions