Add indexed, timestamped, cursor-paginated instance listing APIs

[Copilot]

Note: This PR was re-implemented on a fresh branch from current main (the original branch was ~87 commits stale and conflicting). The branch history was force-updated. The obsolete df.grant_usage() allowlist edit was dropped (that allowlist was removed in 0.2.4, #242, so ordinary df.* functions are now gated solely by schema USAGE), and the new_backend_provider() call was fixed to pass the provider schema.

Motivation

df.instances listing was expensive for external clients because chronological reads had no supporting index, df.list_instances had no server-side pagination, and timestamps were omitted from the response. This made efficient page-by-page UI listing and metadata display harder than necessary.

Changes

• Schema / index

  • Added a chronological index for stable keyset scans: idx_instances_created_at_desc_id on df.instances(created_at DESC, id).

• df.list_instances contract update

  • Extended the return shape to include created_at and completed_at.
  • Switched ordering to ORDER BY created_at DESC, id DESC for a deterministic total order.

• New paginated API

  • Added df.list_instances_paginated(status_filter, limit_count, after_cursor).
  • Keyset pagination on (created_at DESC, id DESC) with a limit + 1 lookahead.
  • Returns the instance fields + timestamps, total_count, and next_cursor (NULL when exhausted).
  • Honors status_filter under existing RLS behavior.

• Upgrade & migration

  • Added DDL to sql/pg_durable--0.2.3--0.2.4.sql (the active unreleased cycle): the index, a drop/recreate of df.list_instances for its new return shape, and the new df.list_instances_paginated binding.
  • Verified Scenario A (fresh-install vs. upgrade schema equivalence) and B1/B2 (binary/data backward compat) via scripts/test-upgrade.sh — 36/36 pass.
  • Documented the change under "Version-Specific Changes" in docs/upgrade-testing.md.
  • Allowlisted the standalone-upgrade-script PS014 finding for the index in scripts/run-pgspot.sh (a plain btree index with no user-defined functions; same false-positive class as the existing PS002 entries).

• Docs

  • Updated USER_GUIDE.md and CHANGELOG.md for the new/expanded monitoring surface.

Testing

• New E2E test tests/e2e/sql/50_list_instances_paginated.sql: starts instances, pages through all of them with a small page size, and asserts the paginated output matches df.list_instances() order exactly with correct total_count and next_cursor semantics (no gaps or duplicates).
• Existing list_instances consumers still pass (05_monitoring_and_explain, 22_cancel_status_consistency).
• cargo fmt, cargo clippy, the upgrade suite, and the pgspot gate all pass.

Usage

-- First page
SELECT * FROM df.list_instances_paginated(NULL, 20, NULL);

-- Next page: pass the previous page's next_cursor as after_cursor
SELECT * FROM df.list_instances_paginated(
  NULL,
  20,
  '2026-01-01 12:00:00+00|a1b2c3d4'
);

Notes for reviewers

• list_instances_paginated is a separate function rather than folding pagination into list_instances, to avoid changing the existing function's argument signature.
• Both functions retain the original per-row get_instance_info lookup (unchanged from existing list_instances behavior; not optimized here).
• The cursor is the non-opaque created_at::text || '|' || id; it round-trips through the timestamptz keyset predicate (validated by the E2E test) but is not an opaque token.

[pinodeca]

Consider combining this PR and #186.