metrics: migrate TiFlash Summary dashboard generation to grafanalib - #11027
Merged
ti-chi-bot[bot] merged 50 commits intoAug 10, 2026
Merged
Conversation
Define the dashboard in jsonnet (split by row) and regenerate tiflash_summary.json so the large JSON is no longer hand-maintained.
Preserve original gridPos so panels stay side-by-side within rows, and emit hide=true for queries that were hidden in the source dashboard.
Rebuild each panel's axes with resetYaxes/addYaxis so label/show/logBase/min match the original dashboard.
Auto-compute panel gridPos from equal-width bands, normalize most panel heights to 8, and keep Storage internal-task bands at height 5.
Centralize max/p9999/p999/p99 targets and shared panel style, and migrate the first set of *_seconds_bucket duration panels to use it.
…anels. Add a hidden avg (sum/count) series to the shared quantile set, then switch remaining single-metric duration panels onto durationPanel for consistency.
Introduce promql.libsonnet (sumRate/sum/histogramQuantile/histogramAvg) and route durationPanel through it so later panel helpers can reuse the same selector-aware expression builders.
Introduce common.target/graph/override helpers and rebuild durationPanel on top of them so row authors can share the same legend and y-axis defaults.
Regenerate the dashboard artifact emptied by a failed jsonnet run and keep the L2 target/graph/override helpers intact.
Introduce common.opsPanel on top of graph/sumRate and convert near- isomorphic single-metric OPS/QPS panels across raft, storage, coprocessor, and related rows.
Introduce common.heatmap for tsbuckets spectral panels and convert single-target raft/snapshot/pagestorage/rough-set heatmaps onto it.
Introduce a helper for proxy thread CPU graphs with a Limit series override, and migrate matching panels in threads_cpu, pipeline, and imbalance rows.
Introduce a helper for OPS series plus hit-ratio overlays on the right axis, and migrate Remote Cache, MVCCIndexCache, and Data Sharing panels.
Support unit/quantiles/showAvg/extraTargets/seriesOverrides on durationPanel, and migrate schema/raft/pipeline/read-pool Class C duration graphs onto the shared helper.
Add an overview table for layout/PromQL/graph/domain helpers so new panels can follow the shared API instead of copying graphPanel boilerplate.
Replace codegen-style names (cPU_UsageP) with identifiers like panelCpuUsage / panelRequestQps / panelStaleReadOps for consistent jsonnet authoring.
Merge rows_*.libsonnet into the entry jsonnet in dashboard order using scoped locals, and drop the per-row fragment files to reduce churn. Keep tiflash_summary.jsonnet/json as the dashboard entry artifacts and point imports/docs at metrics/grafana/tiflashnet for shared helpers.
Replace hand-written sum(...) / sum(rate(...)) queries with common.target(common.expr.sum/sumRate) across TiFlash Summary rows, keeping generated JSON semantically equivalent.
Cover decimals, pointradius, stack, linewidth, legend hide flags, and extra axis min/max/decimals so hand-written graphPanel.new panels can migrate cleanly.
Replace repeated graphPanel.new + resetYaxes/addYaxis boilerplate with common.graph across TiFlash Summary rows; leave one-off panels that need non-table legends, axis labels, or log scales as-is.
Escape PromQL regex \d for background CPU panels, migrate SST Import Service to cpuWithLimitPanel with hideLimit, and drop the fixed legend sideWidth from that helper.
Default common.graph to yRightShow=false, show the right axis only for dual-axis panels, add sumIrate, and rewrite Memory / IO Throughput queries via common.expr.
Expose tooltipSort on common.graph and set decreasing order for OPS and duration helpers so shared tooltips list series from high to low.
Replace remaining fixed [1m] ranges with \$__rate_interval, and remove explicit yRight='short' where the right axis stays hidden by default.
Align with CSE/TiKV Python dashboard tooling to drop the go-jsonnet dependency, and archive the previous jsonnet sources under jsonnet_legacy for a later cleanup.
Restore All/Write/Compute label:value options and default current=.* so Role filters match instances on dashboard open.
Replace underscore-separated RowPanel function names (e.g. Disaggregated_Write) with consistent PascalCase identifiers.
Rely on equal-split defaults and half_row for half-width bands so dashboard authors no longer repeat height=8/widths=... on every band.
Collapse left/right min, label, decimals, logBase, and right-axis visibility into yaxes kwargs so dashboard panels no longer need extra_json or y_right_show.
Put title/description/targets/yaxes first and keep fill next to fill_gradient in dashboard panels and helpers.
ti-chi-bot
Bot
added
release-note-none
📝 WalkthroughWalkthroughThe PR adds a Python/grafanalib library for TiFlash Summary dashboards, generation tooling, documentation, and checksum tracking. It also updates the DMFile V3 design RFC and enables foreign-key checks in two TiDB test configurations. ChangesGrafana dashboard generation
DMFile V3 design update
TiDB test configuration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Developer
participant generate_dashboard.sh
participant uv
participant common.py
participant tiflash_summary.json
participant tiflash_summary.json.sha256
Developer->>generate_dashboard.sh: start generation
generate_dashboard.sh->>uv: synchronize and format environment
generate_dashboard.sh->>common.py: generate dashboard definitions
common.py->>tiflash_summary.json: write generated JSON
generate_dashboard.sh->>tiflash_summary.json.sha256: write SHA-256 checksum
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (7)
docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md (2)
17-17: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winClarify the optional member and size invariants for DMFile V3.
The background says
x.idxandx.null.mrkare optional, but the merge rule treats all three members as present. State that the merged file is built from existing members only, and define the writer behavior when any member exceeds the pre-merge expectation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md` at line 17, Update the DMFile merge-rule description to state that the merged file includes only members that exist, including optional x.idx and x.null.mrk. Define the writer’s behavior when any member exceeds the expected pre-merge size threshold, preserving the stated size invariants for normal files.
19-19: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winClarify DMFile V3 data-file handling.
The writer uses one merged file and stores filename-to-number/offset/size metadata, so readers can locate index, mark, and small data members and support rollover when
merged_file.max_sizeis reached. The RFC still says “small files of each column collectively,” which contradicts the writer behavior and the other recent docs that say index, mark, and small data for all columns are merged together. Add this behavior explicitly or remove the “each column” wording.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md` at line 19, Update the DMFile V3 data-file description to state that index, mark, and small data members across all columns are merged into one file, with filename-to-number/offset/size metadata used for reader lookup and rollover when merged_file.max_size is reached; remove the “each column” wording.metrics/grafana/generate_dashboard.sh (2)
27-28: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote the Python glob so a leading-dash filename cannot become an option. Shellcheck reports SC2035 for the bare
*.pyargument. The same command is copied into the manual instructions, so both places need the same edit.
metrics/grafana/generate_dashboard.sh#L27-L28: change*.pyto./*.pyin theisortandblackinvocations.metrics/grafana/README.md#L24-L25: change*.pyto./*.pyin the two manual commands so the documentation matches the script.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/generate_dashboard.sh` around lines 27 - 28, Update the isort and black commands in metrics/grafana/generate_dashboard.sh at lines 27-28 to use ./*.py instead of *.py, and make the same replacement in the manual commands in metrics/grafana/README.md at lines 24-25 so both script and documentation safely handle leading-dash filenames.Source: Linters/SAST tools
30-40: 📐 Maintainability & Code Quality | 🔵 TrivialOptional: add a
--checkmode for CI.The design document lists CI staleness detection as an unresolved question (
docs/design/2026-08-08-grafanalib-dashboard-generation.mdLine 177). A--checkflag that regenerates into a temporary file and compares the digest would close it without a separate script.Do you want me to generate that flag, or open a new issue to track it?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/generate_dashboard.sh` around lines 30 - 40, Do not modify the current generation script for this optional follow-up; leave the existing dashboard generation and checksum behavior unchanged. Track adding a --check mode separately if CI staleness detection is desired.metrics/grafana/common.py (2)
1545-1559: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffThe
max_hackbranch duplicates selector composition outsideExpr.This branch builds the label set by hand from
CLUSTER_LABEL_SELECTORS,instance_selector, andextra_label_selectors.Expr.__str__performs the same composition at Lines 180-197, plus the shared-pool filtering at Lines 185-190.The rendered PromQL is correct today. It will drift silently if the selector rules change in
Expr. The design document names centralized PromQL as the main goal of this migration.Consider expressing the
1e9rounding throughExpr.extraor a small dedicated builder, so one code path owns selector composition.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/common.py` around lines 1545 - 1559, Refactor the max_hack branch in the query-building function to reuse Expr’s centralized selector composition, including its shared-pool filtering, instead of manually combining CLUSTER_LABEL_SELECTORS, instance_selector, and extra_label_selectors. Express the 1e9 rounding through Expr.extra or a focused builder while preserving the existing histogram_quantile PromQL output.
156-161: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse
factory=listfor the attrs list attributes.
attr.ib(default=[])shares one list across allExprinstances. No code mutates these lists in place today, so there is no live bug. Any future.append()would corrupt every expression in the module.Ruff also reports B006 for the same pattern on about 40 function signatures in this file. A single pass to
Nonedefaults orfactory=listclears the whole class of warnings.♻️ Proposed change for the attrs attributes
- label_selectors: list[str] = attr.ib(default=[], validator=instance_of(list)) - by_labels: list[str] = attr.ib(default=[], validator=instance_of(list)) + label_selectors: list[str] = attr.ib(factory=list, validator=instance_of(list)) + by_labels: list[str] = attr.ib(factory=list, validator=instance_of(list))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/common.py` around lines 156 - 161, Replace the mutable default `default=[]` in the attrs fields `label_selectors` and `by_labels` with `factory=list`, preserving their `list` validators and per-instance empty-list behavior. Also update the approximately 40 function signatures in this module that use mutable list defaults, replacing them with `None` handling or `factory=list` as appropriate to clear Ruff B006 warnings without changing behavior.Source: Linters/SAST tools
metrics/grafana/pyproject.toml (1)
5-10: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove
blackandisortinto thedevdependency group.
generate_dashboard.shneedsuv syncforattrs,grafanalib,black, andisort. Keep onlyattrsandgrafanalibin[project.dependencies], and useuv sync --only-group devingenerate_dashboard.shso dashboard generators do not install unnecessary formatters by default.♻️ Proposed change
dependencies = [ "attrs", "grafanalib==0.7.1", - "black==26.5.1", - "isort==8.0.1", ] + +[dependency-groups] +dev = [ + "black==26.5.1", + "isort==8.0.1", +]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/pyproject.toml` around lines 5 - 10, Move black and isort out of the main dependencies list, keeping only attrs and grafanalib under project.dependencies; declare both formatters in the dev dependency group and update generate_dashboard.sh to run uv sync --only-group dev so its generator setup still installs all required tools.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md`:
- Line 21: Make the downgrade section around DTTool concrete by adding a DMFile
source/target version matrix, explicitly covering merged data files for direct
downgrades and compacted outputs, and documenting the failure behavior for
unsupported layouts. Add a requirement for DMFile read/write round-trip tests
before describing downgrade support as available.
In `@metrics/grafana/common.py`:
- Around line 77-108: Make the documented selector context functional by
updating the Expr factories in metrics/grafana/common.py (lines 77-108) to read
_active_promql_policy instead of module constants, preserving
instance_selectors_policy, use_instance_selectors, and use_promql_policy; then
update metrics/grafana/README.md (lines 45-50) to accurately document the
working behavior. Alternatively, remove the entire unused policy/context-manager
API and remove the use_instance_selectors(...) sentence from the README.
- Around line 733-757: Update expr_topk to honor instance_selector for string
metric inputs by applying the selector to the generated aggregation expression,
while preserving existing behavior for Expr inputs. Ensure the rendered topk
query includes the caller-provided cluster and instance filters instead of
always passing an empty selector and clearing default selectors.
- Around line 1216-1247: Update heatmap_panel so the default YAxis is created
per invocation rather than shared through the signature default. Rename the
parameter to y_axis to avoid shadowing the yaxis factory, initialize y_axis with
yaxis(UNITS.NO_FORMAT) when omitted, and apply decimals only to that call’s axis
object.
In `@metrics/grafana/generate_dashboard.sh`:
- Around line 35-38: Update the checksum-generation block in
generate_dashboard.sh to select an available SHA-256 utility before calculating
the digest, using the platform-appropriate alternative when sha256sum is
unavailable. Ensure the checksum file is only retained or replaced after
successful generation so a missing tool cannot leave a stale
tiflash_summary.json.sha256 sidecar.
---
Nitpick comments:
In `@docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md`:
- Line 17: Update the DMFile merge-rule description to state that the merged
file includes only members that exist, including optional x.idx and x.null.mrk.
Define the writer’s behavior when any member exceeds the expected pre-merge size
threshold, preserving the stated size invariants for normal files.
- Line 19: Update the DMFile V3 data-file description to state that index, mark,
and small data members across all columns are merged into one file, with
filename-to-number/offset/size metadata used for reader lookup and rollover when
merged_file.max_size is reached; remove the “each column” wording.
In `@metrics/grafana/common.py`:
- Around line 1545-1559: Refactor the max_hack branch in the query-building
function to reuse Expr’s centralized selector composition, including its
shared-pool filtering, instead of manually combining CLUSTER_LABEL_SELECTORS,
instance_selector, and extra_label_selectors. Express the 1e9 rounding through
Expr.extra or a focused builder while preserving the existing histogram_quantile
PromQL output.
- Around line 156-161: Replace the mutable default `default=[]` in the attrs
fields `label_selectors` and `by_labels` with `factory=list`, preserving their
`list` validators and per-instance empty-list behavior. Also update the
approximately 40 function signatures in this module that use mutable list
defaults, replacing them with `None` handling or `factory=list` as appropriate
to clear Ruff B006 warnings without changing behavior.
In `@metrics/grafana/generate_dashboard.sh`:
- Around line 27-28: Update the isort and black commands in
metrics/grafana/generate_dashboard.sh at lines 27-28 to use ./*.py instead of
*.py, and make the same replacement in the manual commands in
metrics/grafana/README.md at lines 24-25 so both script and documentation safely
handle leading-dash filenames.
- Around line 30-40: Do not modify the current generation script for this
optional follow-up; leave the existing dashboard generation and checksum
behavior unchanged. Track adding a --check mode separately if CI staleness
detection is desired.
In `@metrics/grafana/pyproject.toml`:
- Around line 5-10: Move black and isort out of the main dependencies list,
keeping only attrs and grafanalib under project.dependencies; declare both
formatters in the dev dependency group and update generate_dashboard.sh to run
uv sync --only-group dev so its generator setup still installs all required
tools.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ea27986c-957e-42f7-be71-655cf7471ff8
⛔ Files ignored due to path filters (1)
metrics/grafana/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.mddocs/design/2026-08-08-grafanalib-dashboard-generation.mdmetrics/grafana/.gitignoremetrics/grafana/README.mdmetrics/grafana/common.pymetrics/grafana/generate_dashboard.shmetrics/grafana/pyproject.tomlmetrics/grafana/tiflash_summary.dashboard.pymetrics/grafana/tiflash_summary.jsonmetrics/grafana/tiflash_summary.json.sha256
Exclude generated sha256 and uv.lock from license-header checks.
Remove unused PromQLPolicy/use_instance_selectors, honor instance_selector in expr_topk, and avoid sharing Heatmap YAxis defaults via per-call y_axis.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
metrics/grafana/common.py (1)
1188-1189: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCreate
Tooltipand heatmap color objects per call.Lines 1188-1189 evaluate these defaults once when Python defines
heatmap_panel. Each default call can therefore pass the same mutable objects to multiple panels. Use sentinels orNonedefaults, then create the objects in the function body.Proposed fix
+_DEFAULT_HEATMAP_TOOLTIP = object() +_DEFAULT_HEATMAP_COLOR = object() + def heatmap_panel( ... - tooltip=Tooltip(shared=True, valueType="individual"), - color=heatmap_color(), + tooltip=_DEFAULT_HEATMAP_TOOLTIP, + color=_DEFAULT_HEATMAP_COLOR, ... ) -> Panel: if y_axis is None: y_axis = yaxis(UNITS.NO_FORMAT, decimals=1) + if tooltip is _DEFAULT_HEATMAP_TOOLTIP: + tooltip = Tooltip(shared=True, valueType="individual") + if color is _DEFAULT_HEATMAP_COLOR: + color = heatmap_color()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@metrics/grafana/common.py` around lines 1188 - 1189, Update the heatmap_panel function defaults so Tooltip and heatmap_color objects are not created at definition time; use None or sentinel defaults, then instantiate fresh objects inside the function body for each call while preserving explicitly supplied values.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@metrics/grafana/common.py`:
- Around line 1188-1189: Update the heatmap_panel function defaults so Tooltip
and heatmap_color objects are not created at definition time; use None or
sentinel defaults, then instantiate fresh objects inside the function body for
each call while preserving explicitly supplied values.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e3939ebe-fdc3-488a-b003-7781ccbdfaf3
📒 Files selected for processing (5)
.github/licenserc.ymlmetrics/grafana/README.mdmetrics/grafana/common.pymetrics/grafana/pyproject.tomlmetrics/grafana/tiflash_summary.dashboard.py
💤 Files with no reviewable changes (1)
- metrics/grafana/README.md
Signed-off-by: JaySon-Huang <tshent@qq.com>
Contributor
Author
Why Python dashboards help coding agentsDefining Grafana dashboards in Python (via grafanalib + shared helpers) is also more friendly for coding agents than maintaining raw JSON:
Caveat: agents still need Grafana/grafanalib semantics, and plugin-specific options may still require |
ti-chi-bot
Bot
added
needs-1-more-lgtm
JinheLin
approved these changes
Aug 10, 2026
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: JinheLin, windtalker The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
lgtm
and removed
needs-1-more-lgtm
Contributor
[LGTM Timeline notifier]Timeline:
|
Signed-off-by: JaySon-Huang <tshent@qq.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/docker/next-gen-columnar-config/tidb.toml`:
- Around line 41-42: Remove the duplicate [experimental] table header while
retaining allow-enable-foreign-key-check-in-shared-lock = true under the
existing table in tests/docker/next-gen-columnar-config/tidb.toml lines 41-42
and tests/docker/next-gen-config/tidb.toml lines 37-38; each file must contain
only one [experimental] header.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 68b6d601-048f-4a3f-a810-6881b7efa99b
📒 Files selected for processing (2)
tests/docker/next-gen-columnar-config/tidb.tomltests/docker/next-gen-config/tidb.toml
Contributor
Author
|
/test pull-integration-test |
Contributor
Author
|
/cherry-pick release-nextgen-202603 |
12 tasks
Member
|
@JaySon-Huang: new pull request created to branch DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
ti-chi-bot Bot
pushed a commit
that referenced
this pull request
Aug 11, 2026
…11027) (#11033) close #11025, close #11028\n\nmetrics: migrate TiFlash Summary dashboard generation to grafanalib Replace hand-maintained TiFlash Summary JSON with a grafanalib pipeline (`common.py` + `tiflash_summary.dashboard.py`), shared PromQL/panel helpers, and `generate_dashboard.sh` to produce `tiflash_summary.json`.\n\nSigned-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>\n\nCo-authored-by: JaySon <tshent@qq.com>
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #11025, close #11028
Problem Summary:
TiFlash Summary Grafana dashboard was maintained as a large hand-edited JSON
blob, which made panel authoring, PromQL consistency, and review hard. This PR
migrates generation to grafanalib (same pattern as TiKV:
common.py+*.dashboard.py), so the dashboard source is Python andtiflash_summary.jsonis a generated artifact. We deliberately prefer Python/
uvover introducing aGo/jsonnet toolchain into TiFlash’s developer workflow and CI.
Design:
docs/design/2026-08-08-grafanalib-dashboard-generation.mdWhat is changed and how it works?
metrics/grafana/common.py: PromQL builders (expr_*),graph_panel/yaxes/Layout, and L3 helpers (ops_panel,duration_panel,cpu_with_limit_panel, heatmap / hit-ratio helpers).tiflash_summary.dashboard.pyas the editable source of truth forTiFlash Summary.
tiflash_summary.json(+.sha256) viagenerate_dashboard.sh(
uv+ grafanalib 0.7.1).metrics/grafana/README.md.docs/design/2026-08-08-grafanalib-dashboard-generation.md.[$__rate_interval]over fixed ranges like[1m]/[30s].right_formaton single-axis panels.yaxis()(Threads IO:Bps→binBps).cpu_with_limit_paneland include$proxy_instance.Check List
Tests
Manual test:
cd metrics/grafana && ./generate_dashboard.shtiflash_summary.json/.sha256are updated and review the diff.Server, Threads CPU, Threads, Coprocessor, Columnar Storage, Raft.
scripts/compare_dashboards.pyfor semanticJSON compare against a legacy baseline. After verification that helper was
removed from tree; it can be restored from git history if needed, e.g.
git show <commit>:metrics/grafana/scripts/compare_dashboards.py.Side effects
Notes: Generated dashboard JSON may differ cosmetically from the previous export
(schema metadata, hidden right-axis defaults, rate interval variable). PromQL /
panel coverage is intended to stay equivalent aside from the intentional
alignments listed above.
tiflash_proxy_summary.json/tiflash_proxy_details.jsonremain hand-maintained.Documentation
Release note
Summary by CodeRabbit
Documentation
New Features