Skip to content

storage: test that checkWriteRequest does not format metric families - #866

Merged
jan--f merged 1 commit into
prometheus:masterfrom
KR-Ravindra:fix/nop-logger-formats-metric-families
Oct 8, 2026
Merged

jan--f merged 1 commit into
prometheus:masterfrom
KR-Ravindra:fix/nop-logger-formats-metric-families

Conversation

@KR-Ravindra

@KR-Ravindra KR-Ravindra commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Since Pushgateway 1.11 switched to promslog, push latency grows with the size of the store as soon as any stored metric family has an inconsistent help string, even with --log.level=warn (#863: ~63% of CPU in slog.(*TextHandler).Handle under checkWriteRequest).

checkWriteRequest runs the consistency check on a scratch DiskMetricStore with promslog.NewNopLogger(). Up to prometheus/common v0.70.x that logger is a TextHandler writing to io.Discard whose Enabled() returns true, so the Info calls in processWriteRequest and GetMetricFamilies that pass whole *dto.MetricFamily values as attributes serialize every metric of the affected families on every push.

Fix

prometheus/common v0.71.0 (prometheus/common#960) makes NewNopLogger return a disabled slog.DiscardHandler. master picked that version up in #871, so this PR is now only the regression test: TestCheckWriteRequestInconsistentHelpAllocs pushes one metric against a stored 1000-metric family and asserts that an inconsistent help string does not cost more allocations than a consistent one. It fails on v0.69.0 (34115 extra allocations) and passes on master.

Fixes #863

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

@KR-Ravindra
KR-Ravindra marked this pull request as ready for review September 14, 2026 07:51
@KR-Ravindra

Copy link
Copy Markdown
Contributor Author

Two weeks without a reviewer, so a short ping. This bumps prometheus/common to v0.71.0, which includes prometheus/common#960 where promslog.NewNopLogger returns slog.New(slog.DiscardHandler) instead of a handler that formats records and throws them away — pushgateway builds a nop logger per discarded push, so the formatting cost was real and measurable. Build and tests are green. Anything I can do to move it along?

@KR-Ravindra
KR-Ravindra force-pushed the fix/nop-logger-formats-metric-families branch from 46e678b to b9ddd8b Compare October 6, 2026 10:29
checkWriteRequest runs the consistency check on a scratch DiskMetricStore
with promslog.NewNopLogger(). Up to prometheus/common v0.70.x that logger
was a TextHandler writing to io.Discard whose Enabled() returned true, so
the Info calls in processWriteRequest and GetMetricFamilies that pass
whole *dto.MetricFamily values as attributes serialized every metric of
the affected families on every push, even with --log.level=warn. Push
latency therefore scaled with the size of the store whenever any stored
metric family had an inconsistent help string.

prometheus/common v0.71.0 (prometheus/common#960) makes NewNopLogger
return a disabled slog.DiscardHandler; master picked that version up in
prometheus#871. Add a regression test that checks a push against a 1000-metric
family with an inconsistent help string does not allocate more than a
push with a consistent one.

Signed-off-by: KR Ravindra <42912207+KR-Ravindra@users.noreply.github.com>
@KR-Ravindra
KR-Ravindra force-pushed the fix/nop-logger-formats-metric-families branch from b9ddd8b to da4bc00 Compare October 6, 2026 10:29
@KR-Ravindra KR-Ravindra changed the title Bump prometheus/common to v0.71.0 to stop formatting discarded logs storage: test that checkWriteRequest does not format metric families Oct 6, 2026

@jan--f jan--f left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@jan--f
jan--f merged commit 3f1be2f into prometheus:master Oct 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

performance: High latency increase after upgrading from 1.10 to 1.11

2 participants