Skip to content

Web viewer: a read that times out returns an empty HTTP 500 and writes nothing to the service log #4276

Description

@erikdarlingdata

What happens

When a web viewer read runs past the viewer role's statement_timeout (15 seconds), PostgreSQL cancels it. The endpoint returns HTTP 500 with an empty body, and the service log has no line for it. The only trace is in the PostgreSQL log: ERROR: canceling statement due to statement timeout.

Seen on a field store on 2026-09-25: GET /api/ag returned 500 after 15.65 seconds. Several page loads were reading /api/ag at the same moment. A single warm call takes 1.4 to 2.2 seconds. The repeated page-load reads are fixed separately (#4189's count probe, #4228). This issue is about any web read that fails.

Why there is no log line

DarlingWebHostService clears the web host's log providers on purpose, so request noise stays out of the service log. An endpoint that needs to log is meant to use the service logger that MapAll receives.

But /api/ag, /api/fleet and some other routes have no try/catch. Their exceptions reach ASP.NET Core's own error logging. That logging writes to the cleared providers, so the line is lost. The browser gets ASP.NET Core's default empty 500.

What to change

  • Add one error handler at the top of the web pipeline, so every /api/* route gets it without its own try/catch.
    • It logs the route, the elapsed time, and the kind of failure through the service logger. A statement timeout (SQLSTATE 57014) is its own kind, apart from other errors.
    • A browser that closed the page (RequestAborted) is not an error, and it is not logged.
    • It returns a JSON error body the viewer can show. For a timeout: "The store took longer than 15 seconds to answer. Try again."
    • A timeout gets a status the viewer can tell apart from a bug, for example 503. Check how the viewer's fetch helper treats that status first.
  • The viewer shows that message in the panel, not a generic failure.
  • Check that the /api/read/* routes write a log line too. They already catch exceptions and return an error envelope.
  • Tests:
    • A route that throws a 57014 PostgresException returns the JSON body and writes one log line.
    • A browser abort writes no log line.
    • A source pin that no /api/* route is mapped outside the handler.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

Activity

  1. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Ruled: how this gets built

    • One error handler at the top of the web pipeline. It logs through the service's own logger, because the web host's logger writes nowhere by design.
    • A statement timeout returns 503 with a JSON message: the store took too long to answer this read, try again in a moment. That covers SQLSTATE 57014 and a client-side command timeout. Any other failure returns 500 with a generic message, never the exception text.
    • The JSON body reuses the shape the web endpoints already return for errors, so the viewer's existing error display can show it.
    • One log line per failed request: a warning for a timeout, an error for anything else. It names the route, the elapsed time, the kind, and the exception type. There is no throttle.
    • A browser that leaves the page is not an error and writes nothing.
    • The /api/read/* routes get a log line too, if they do not write one today.
    • Lite has no web host, so nothing changes there.

    \U0001F916 Generated with Claude Code

    https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3

  2. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  3. added 2 commits that reference this issue on Sep 25, 2026
  4. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #4281, merged to dev (baaa4e0).

  5. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions