Skip to content

feat: add stream consumer group backlog report - #74

Open
SahilRajputdotexe wants to merge 1 commit into
HDT3213:masterfrom
SahilRajputdotexe:feat/stream-backlog-report
Open

SahilRajputdotexe wants to merge 1 commit into
HDT3213:masterfrom
SahilRajputdotexe:feat/stream-backlog-report

Conversation

@SahilRajputdotexe

Copy link
Copy Markdown

This adds a streambacklog command that reports how far each stream consumer group is behind, read straight from an RDB file.

A group keeps its pending entries as StreamNAck values (id, delivery time, delivery count) and each consumer keeps its own list of pending ids, but nothing joined the two or compared message ids against the group's LastId. This does that join in model and writes the result as CSV from helper:

  • StreamGroup.PendingOwners() maps each pending id to the sorted consumers that hold it.
  • StreamObject.UndeliveredMessages(group) returns the live messages above the group's LastId that the group has no pending entry for, ordered by id. It compares every message's own id, since a stream entry holds many messages.
  • StreamObject.ConsumerBacklog() returns one row per consumer with pending count, oldest delivery time and max delivery count, plus the group's undelivered count and the number of dangling pending entries (message deleted or gone). Pending entries nobody holds go into one row with an empty consumer, and a group with no pending entries still gets a row.

Usage:

rdb -c streambacklog [-o backlog.csv] dump.rdb
database,key,group,consumer,pending,oldest_delivery_time,max_delivery_count,undelivered,dangling
0,listpack,g1,c1,2,1528516636879,1,144,0
0,listpack,g4,,0,0,0,0,0

It takes the same options as the other reports (-regex etc.). I added the command to both READMEs.

Tests are table-driven unit tests for the three model methods, plus report tests against cases/stream_listoacks_3.rdb and stream_listpacks_1.rdb, and an encoder-built file covering CSV quoting, key order across databases and non-stream keys being skipped. go test ./... passes in the Go workflow on my fork.

Add StreamGroup.PendingOwners, StreamObject.UndeliveredMessages and
StreamObject.ConsumerBacklog to relate a group's pending entries to the
consumers holding them, and a streambacklog command that writes the
result as CSV.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant