diff --git a/.github/licenserc.yml b/.github/licenserc.yml index effe919be27..6ae4a3935fa 100644 --- a/.github/licenserc.yml +++ b/.github/licenserc.yml @@ -48,5 +48,7 @@ header: - 'OWNERS_ALIASES' - '**/*.sql' - 'dbms/src/Common/tests/tls/' + - 'metrics/grafana/tiflash_summary.json.sha256' + - 'metrics/grafana/uv.lock' comment: on-failure diff --git a/docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash b/docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md similarity index 96% rename from docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash rename to docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md index d8ea322c95d..1966d778075 100644 --- a/docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash +++ b/docs/design/2023-08-04-reduce-the-number-of-files-used-by-TiFlash.md @@ -12,10 +12,10 @@ Currently, for each section in DMFile we generate a separate file. Specifically, Therefore, we consider merging the small files in DMFiles to cut down the number of inode and enhance TiFlash availability. -## Detailed Desgin +## Detailed Design For these files in DMFiles, `x.idx`, `x.null.mrk`, `x.mrk` are always very small, stablely less than 4KB. Besides, when the table contains tiny data, the file sizes of `x.dat` and `x.null.dat` are also very small. Therefore, for each DMFile, we can always merge `x.idx`, `x.null.mrk` and `x.mrk` together. For `x.dat` and `x.null.dat`, we can decide whether they should be merged based on their actual file size with our min file size threshold. Furthermore, we do not merge the files of each column individually, but merge all the small files of each column collectively. In order to avoid the merged file being too large, we will set a max file size threshold. When the merged file reaches the threshold, we will close this merged file and write it into the next merged file. -When upgrade the TiFlash version, we directly support the upgraded TiFlash to read the old version DMFiles and write into new version DMFiles when do compaction later. To downgrade the TiFlash version, we support a `DTTool` to support rewriting the new version DMFiles to old versions offline. \ No newline at end of file +When upgrade the TiFlash version, we directly support the upgraded TiFlash to read the old version DMFiles and write into new version DMFiles when do compaction later. To downgrade the TiFlash version, we support a `DTTool` to support rewriting the new version DMFiles to old versions offline. diff --git a/docs/design/2026-08-08-grafanalib-dashboard-generation.md b/docs/design/2026-08-08-grafanalib-dashboard-generation.md new file mode 100644 index 00000000000..986fcc7dac4 --- /dev/null +++ b/docs/design/2026-08-08-grafanalib-dashboard-generation.md @@ -0,0 +1,179 @@ +# Generate TiFlash Grafana Dashboards with Python grafanalib + +- Author(s): [JaySon-Huang](https://github.com/JaySon-Huang) + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation or Background](#motivation-or-background) +* [Detailed Design](#detailed-design) +* [Test Design](#test-design) +* [Impacts & Risks](#impacts--risks) +* [Investigation & Alternatives](#investigation--alternatives) +* [Unresolved Questions](#unresolved-questions) + +## Introduction + +This design migrates **TiFlash Summary** (and sets the pattern for other TiFlash Grafana dashboards) from hand-maintained to a **Python + [grafanalib](https://github.com/weaveworks/grafanalib)** generation pipeline, aligned with TiKV (`common.py` + `*.dashboard.py`). +Dashboard authors edit Python sources; checked-in `.json` is a generated artifact produced by `generate_dashboard.sh`. + +## Motivation or Background + +### Problems with the previous approach + +1. **Hard to review and evolve**: TiFlash Summary lived as a very large Grafana + JSON export. Small panel changes produced noisy diffs; PromQL strings were + duplicated with inconsistent label selectors and rate windows. +2. **Weak abstraction**: Common patterns (OPS `sum(rate)`, duration histograms, + Threads CPU + Limit, heatmaps) were copy-pasted instead of shared helpers. +3. **Tooling mismatch**: Intermediate jsonnet/grafonnet work improved structure + but stayed farther from the TiKV Python tooling that the monitoring + ecosystem already standardizes on. More importantly, we do not want to + introduce a **Go/jsonnet toolchain dependency** into TiFlash’s developer + workflow and CI; Python + `uv` / grafanalib fits the existing metrics + authoring path without expanding the build/test tool surface. + +### Goals + +- Make **Python sources** the source of truth for TiFlash Summary. +- Provide layered helpers so new panels reuse PromQL builders and panel + factories instead of raw JSON. +- Keep generated JSON importable by Grafana / Clinic / TiUP packaging with + stable dashboard `uid` / datasource `__inputs`. +- Prefer intentional, documented semantic alignments (e.g. `$__rate_interval`) + over silent regressions. + +### Non-goals (this phase) + +- Rewriting `tiflash_proxy_summary.json` / `tiflash_proxy_details.json` (remain + hand-maintained for now). +- Changing TiFlash runtime metrics emission or Prometheus scrape config. +- Requiring CI to import dashboards into a live Grafana (optional follow-up). + +## Detailed Design + +### Directory layout + +```text +metrics/grafana/ + common.py # shared PromQL + panel helpers + tiflash_summary.dashboard.py # TiFlash Summary source + tiflash_summary.json # generated (do not edit) + tiflash_summary.json.sha256 + generate_dashboard.sh # uv sync + format + generate + pyproject.toml / uv.lock # grafanalib==0.7.1 + README.md +``` + +During migration we also used a temporary `scripts/compare_dashboards.py` for +semantic JSON diffs against legacy baselines; it is not kept in tree afterward. + +### Generation flow + +Authors run: + +```bash +cd metrics/grafana +./generate_dashboard.sh +``` + +The script syncs the `uv` env, runs `isort`/`black` on `*.py`, then +`generate-dashboard -o tiflash_summary.json tiflash_summary.dashboard.py`, and +updates the SHA256 sidecar. + +### Layered helper model + +The design mirrors CSE’s four-layer DSL: + +```text + Expr / OpExpr + (expr_sum_rate, expr_histogram_*) + │ + v + target() + (legend / hide / interval_factor) + │ + v + graph_panel / yaxes / Layout + │ + v + ops_panel / duration_panel / + cpu_with_limit_panel / heatmap +``` + +1. **PromQL builders** (`Expr`, `expr_sum`, `expr_sum_rate`, histogram helpers): + always attach cluster selectors (`k8s_cluster` / `tidb_cluster`) and choose + instance selectors via `instance_selector`: + - `CPP_LABEL_SELECTORS`: `$instance` + `$tiflash_role` + - `PROXY_LABEL_SELECTORS`: `$proxy_instance` + `$tiflash_role` +2. **`target()`**: wraps PromQL into grafanalib `Target` with legend / hide. +3. **`graph_panel()` / `yaxes()` / `Layout`**: shared visual defaults (legend + table, tooltip sort, single-axis `right_show=False`, IEC byte-unit assert). +4. **Domain panels**: + - `ops_panel`: single `sum(rate(...))` OPS-style graph + - `duration_panel`: S3-style histogram quantiles (max/9999/999/99/80/avg) + - `cpu_with_limit_panel`: Threads CPU series + Limit override + - heatmap / hit-ratio helpers for specialized rows + +Dashboard rows are ordinary Python functions returning `RowPanel`, composed in the dashboard entrypoint. + +### Compatibility + +- **Grafana / Clinic / TiUP**: keep dashboard `uid` and `__inputs` datasource + wiring so existing imports can overwrite the same dashboard identity when + desired. +- **External components**: no change to TiDB / TiKV / PD metrics contracts; + only how TiFlash Summary JSON is authored. + +## Test Design + +### Functional Tests + +- Regenerate with `./generate_dashboard.sh`; confirm exit 0 and SHA256 update. +- Python sources format cleanly under repo `isort`/`black` settings. +- Spot-import generated JSON into a test Grafana and verify datasource binding + (`DS_TEST-CLUSTER` → local Prometheus). + +### Compatibility Tests + +- Semantic compare against a known baseline JSON when available (migration + phase used `scripts/compare_dashboards.py`; not retained after landing). + +### Benchmark Tests + +Not applicable: this change does not affect TiFlash query/storage runtime performance. + +## Impacts & Risks + +### Impacts + +- **Positive**: smaller, reviewable panel diffs; reusable helpers; consistent + PromQL selectors and units; same authoring model as TiKV. +- **Positive**: clearer ownership — edit `.dashboard.py`, never hand-edit + generated `.json`. +- **Neutral / operational**: dashboard JSON shape may differ cosmetically + (schema metadata, default legend flags) while queries remain equivalent aside + from documented alignments. + +### Risks + +- **Silent PromQL drift** if helpers compose selectors incorrectly + (mitigation: migration-time `compare_dashboards.py`, Grafana spot-check, + code review). +- **grafanalib version skew** (`0.7.1` pinned in `uv.lock`); upgrading may + change JSON defaults (mitigation: pin + regenerate in the same PR). + +## Investigation & Alternatives + +| Approach | Pros | Cons | Decision | +|----------|------|------|----------| +| Keep hand-edited JSON | Zero tooling | Unmaintainable diffs | Rejected | +| jsonnet + grafonnet-lib | Structured, used briefly in TiFlash | Diverges from TiKV Python; extra golang toolchain | Rejected as long-term SoT | +| **Python grafanalib** | Matches TiKV; rich helpers; `uv` workflow | Need Python env for regen | **Chosen** | +| Grafana UI only / provisioning CRDs | Nice for ops clusters | Poor fit for git-reviewed upstream metrics | Out of scope | + +## Unresolved Questions + +- Whether CI should fail if `tiflash_summary.json` is stale vs sources (SHA / regenerate check). +- Timeline / ownership for migrating `tiflash_proxy_summary.json` and + `tiflash_proxy_details.json` to the same pipeline. diff --git a/metrics/grafana/.gitignore b/metrics/grafana/.gitignore new file mode 100644 index 00000000000..705c0e53623 --- /dev/null +++ b/metrics/grafana/.gitignore @@ -0,0 +1,5 @@ +.venv +build/ +tiflash_grafana_dashboards.egg-info/ +__pycache__/ +*.pyc diff --git a/metrics/grafana/README.md b/metrics/grafana/README.md new file mode 100644 index 00000000000..9edf886e11b --- /dev/null +++ b/metrics/grafana/README.md @@ -0,0 +1,59 @@ +# TiFlash Grafana dashboards + +TiFlash Summary is generated as Grafana JSON from Python code using +[grafanalib](https://github.com/weaveworks/grafanalib), following the same +pattern as TiKV / Cloud Storage Engine (`common.py` + `*.dashboard.py`). + +Please avoid manually modifying the generated `.json` files. + +## Generate Dashboard JSON + +```bash +cd metrics/grafana +./generate_dashboard.sh +``` + +This runs `uv sync`, formats Python sources with isort/black, regenerates +`tiflash_summary.json`, and updates `tiflash_summary.json.sha256`. + +### Manual (uv) + +```bash +cd metrics/grafana +uv sync +.venv/bin/isort --profile black *.py +.venv/bin/black *.py +.venv/bin/generate-dashboard -o tiflash_summary.json tiflash_summary.dashboard.py +``` + +## Files + +| File | Description | +|------|-------------| +| `common.py` | Shared helpers: PromQL builders, `graph_panel`, L3 panels (`ops_panel`, `duration_panel`, `cpu_with_limit_panel`, …) | +| `tiflash_summary.dashboard.py` | TiFlash Summary dashboard source | +| `tiflash_summary.json` | Generated JSON — do not edit manually | +| `tiflash_summary.json.sha256` | SHA256 of the generated JSON | +| `generate_dashboard.sh` | Generate entrypoint | +| `pyproject.toml` / `uv.lock` | Python deps (`grafanalib==0.7.1`) | + +## Authoring notes + +- Prefer helpers in `common.py` for new panels (`ops_panel`, `duration_panel`, + `tiflash_heatmap_panel`, `cpu_with_limit_panel`, `ops_hit_ratio_panel`, + `graph_panel` + `expr_*`). +- Default PromQL labels always include `k8s_cluster` / `tidb_cluster`. + Choose instance selectors with `instance_selector=CPP_LABEL_SELECTORS` + (default: `$instance` + `$tiflash_role`) or + `instance_selector=PROXY_LABEL_SELECTORS` (`$proxy_instance` + + `$tiflash_role`). Put non-instance filters only in `label_selectors`. +- `tiflash_proxy_summary.json` / `tiflash_proxy_details.json` are still + hand-maintained JSON. + +## Validate + +```bash +cd metrics/grafana +./generate_dashboard.sh +# Confirm tiflash_summary.json / .sha256 are updated and review the diff. +``` diff --git a/metrics/grafana/common.py b/metrics/grafana/common.py new file mode 100644 index 00000000000..52519dbdf3a --- /dev/null +++ b/metrics/grafana/common.py @@ -0,0 +1,1794 @@ +# Copyright 2026 PingCAP, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import re +from typing import Optional, Sequence, Union + +import attr +from attr.validators import in_, instance_of +from grafanalib import formatunits as UNITS +from grafanalib.core import ( + NULL_AS_ZERO, + TIME_SERIES_TARGET_FORMAT, + DataSourceInput, + Graph, + GraphThreshold, + GridPos, + Heatmap, + HeatmapColor, + Legend, + Panel, + RowPanel, + SeriesOverride, + Stat, + StatValueMappings, + Target, + Template, + TimeSeries, + Tooltip, + YAxes, + YAxis, +) + +DATASOURCE_INPUT = DataSourceInput( + name="DS_TEST-CLUSTER", + label="Test-Cluster", + pluginId="prometheus", + pluginName="Prometheus", +) +DATASOURCE = f"${{{DATASOURCE_INPUT.name}}}" +ADDITIONAL_GROUPBY = "$additional_groupby" +OPTIONAL_QUANTILE = "optional_quantile" +OPTIONAL_QUANTILE_INPUT = "$" + OPTIONAL_QUANTILE + +_TIDB_CLUSTER_SELECTOR_RE = re.compile(r"^\s*tidb_cluster(_id)?\s*(=~|=|!=|!~).*$") + +CLUSTER_LABEL_SELECTORS = ( + r'k8s_cluster="$k8s_cluster"', + r'tidb_cluster="$tidb_cluster"', +) +CPP_LABEL_SELECTORS = ( + r'instance=~"$instance"', + r'instance=~"$tiflash_role"', +) +PROXY_LABEL_SELECTORS = ( + r'instance=~"$proxy_instance"', + r'instance=~"$tiflash_role"', +) + + +@attr.s +class Expr(object): + """ + A prometheus expression that matches the following grammar: + + expr ::= ( + [aggr_param,] + [func]( + + [{,}] + [[]] + ) + ) [by (,)] [extra_expr] + """ + + def metric_validator(instance, attribute, value): + if not isinstance(value, (str, Expr)): + raise TypeError( + f"'{attribute.name}' must be an instance of 'str' or 'Expr'" + ) + + metric: Union[str, Expr] = attr.ib(validator=metric_validator) + aggr_op: str = attr.ib( + default="", + validator=in_( + [ + "", + "sum", + "min", + "max", + "avg", + "group", + "stddev", + "stdvar", + "count", + "count_values", + "bottomk", + "topk", + "quantile", + ] + ), + ) + aggr_param: str = attr.ib(default="", validator=instance_of(str)) + func: str = attr.ib(default="", validator=instance_of(str)) + range_selector: str = attr.ib(default="", validator=instance_of(str)) + label_selectors: list[str] = attr.ib(default=[], validator=instance_of(list)) + by_labels: list[str] = attr.ib(default=[], validator=instance_of(list)) + default_label_selectors: list[str] = attr.ib( + factory=lambda: list(CLUSTER_LABEL_SELECTORS), + validator=instance_of(list), + ) + instance_selector: tuple = attr.ib( + factory=lambda: tuple(CPP_LABEL_SELECTORS), + converter=tuple, + ) + shared_pool_selector: Optional[str] = attr.ib(default=None) + use_shared_pool: bool = attr.ib(default=False, validator=instance_of(bool)) + extra_expr: str = attr.ib(default="", validator=instance_of(str)) + + def __str__(self) -> str: + aggr_opeator = self.aggr_op if self.aggr_op else "" + aggr_param = self.aggr_param + "," if self.aggr_param else "" + by_clause = ( + "by ({})".format(", ".join(self.by_labels)) if self.by_labels else "" + ) + func = self.func if self.func else "" + # CLUSTER (+ optional cleared defaults) + instance_selector + extra label_selectors. + label_selectors = ( + list(self.default_label_selectors) + + list(self.instance_selector) + + list(self.label_selectors) + ) + if self.use_shared_pool: + label_selectors = [ + l for l in label_selectors if not _TIDB_CLUSTER_SELECTOR_RE.match(l) + ] + if self.shared_pool_selector is not None: + label_selectors.append(self.shared_pool_selector) + + assert all( + ("=" in item or "~" in item) for item in label_selectors + ), f"Not all items contain '=' or '~', invalid {self.label_selectors}" + instant_selectors = ( + "{{{}}}".format(",".join(label_selectors)) if label_selectors else "" + ) + range_selector = f"[{self.range_selector}]" if self.range_selector else "" + extra_expr = self.extra_expr if self.extra_expr else "" + return f"""{aggr_opeator}({aggr_param}{func}( + {self.metric} + {instant_selectors} + {range_selector} +)) {by_clause} {extra_expr}""" + + def aggregate( + self, + aggr_op: str, + aggr_param: str = "", + by_labels: list[str] = [], + label_selectors: list[str] = [], + ) -> "Expr": + self.aggr_op = aggr_op + self.aggr_param = aggr_param + self.by_labels = by_labels + self.label_selectors = label_selectors + return self + + def function( + self, + func: str, + label_selectors: list[str] = [], + range_selector: str = "", + ) -> "Expr": + self.func = func + self.label_selectors = label_selectors + self.range_selector = range_selector + return self + + def extra( + self, + extra_expr: Optional[str] = None, + default_label_selectors: Optional[list[str]] = None, + ) -> "Expr": + if extra_expr is not None: + self.extra_expr = extra_expr + if default_label_selectors is not None: + self.default_label_selectors = default_label_selectors + return self + + def use_shared_pool_selector(self) -> "Expr": + self.use_shared_pool = True + return self + + def append_by_labels(self, label: str) -> "Expr": + if isinstance(self.metric, Expr): + # append the label to the inner expr + self.metric.append_by_labels(label) + else: + self.by_labels = self.by_labels + [label] + return self + + +class OpExpr: + lhs: Union[Expr, OpExpr, str] + op: str + rhs: Union[Expr, OpExpr, str] + + def __init__( + self, lhs: Union[Expr, OpExpr, str], op: str, rhs: Union[Expr, OpExpr, str] + ): + self.lhs = lhs + self.op = op + self.rhs = rhs + + def __str__(self) -> str: + return f"""({self.lhs} {self.op} {self.rhs})""" + + def __repr__(self) -> str: + return self.__str__() + + def append_by_labels(self, label: str) -> "OpExpr": + if not isinstance(self.lhs, str): + self.lhs.append_by_labels(label) + if not isinstance(self.rhs, str): + self.rhs.append_by_labels(label) + return self + + +def expr_aggr( + metric: Union[str, Expr], + aggr_op: str, + aggr_param: str = "", + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], + use_shared_pool: bool = False, +) -> Expr: + """ + Calculate the aggregation of a metric. + + Example: + + sum(( + tikv_store_size_bytes + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + )) by (instance) + """ + expr = Expr(metric=metric) + expr.aggregate( + aggr_op, + aggr_param=aggr_param, + by_labels=by_labels, + label_selectors=label_selectors, + ) + expr.instance_selector = tuple(instance_selector) + expr.use_shared_pool = use_shared_pool + return expr + + +def expr_sum( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the sum of a metric. + + Example: + + sum(( + tikv_store_size_bytes + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + )) by (instance) + """ + return expr_aggr( + metric, + "sum", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ) + + +def expr_avg( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the avg of a metric. + + Example: + + avg(( + tikv_store_size_bytes + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + )) by (instance) + """ + return expr_aggr( + metric, + "avg", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ) + + +def expr_max( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the max of a metric. + + Example: + + max(( + tikv_store_size_bytes + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + )) by (instance) + """ + return expr_aggr( + metric, + "max", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ) + + +def expr_min( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the min of a metric. + + Example: + + min(( + tikv_store_size_bytes + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + )) by (instance) + """ + return expr_aggr( + metric, + "min", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ) + + +def expr_aggr_func( + metric: str, + aggr_op: str, + func: str, + aggr_param: str = "", + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + range_selector: str = "", + by_labels: list[str] = ["instance"], + use_shared_pool: bool = False, +) -> Expr: + """ + Calculate the aggregation of function of a metric. + + Example: + + expr_aggr_func( + tikv_grpc_msg_duration_seconds_count, "sum", "rate", lables_selectors=['type!="kv_gc"'] + ) + + sum(rate( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (instance) + """ + expr = Expr(metric=metric) + expr.aggregate( + aggr_op, + aggr_param=aggr_param, + by_labels=by_labels, + ) + expr.function( + func, + label_selectors=label_selectors, + range_selector=range_selector, + ) + expr.instance_selector = tuple(instance_selector) + expr.use_shared_pool = use_shared_pool + return expr + + +def expr_sum_rate( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], + use_shared_pool: bool = False, +) -> Expr: + """ + Calculate the sum of rate of a metric. + + Example: + + sum(rate( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (instance) + """ + # $__rate_interval is a Grafana variable that is specialized for Prometheus + # rate and increase function. + # See https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/ + return expr_aggr_func( + metric=metric, + aggr_op="sum", + func="rate", + label_selectors=label_selectors, + range_selector="$__rate_interval", + by_labels=by_labels, + instance_selector=instance_selector, + use_shared_pool=use_shared_pool, + ) + + +def expr_sum_irate( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], + use_shared_pool: bool = False, +) -> Expr: + """sum(irate(metric{...}[$__rate_interval])) by (...)""" + return expr_aggr_func( + metric=metric, + aggr_op="sum", + func="irate", + label_selectors=label_selectors, + range_selector="$__rate_interval", + by_labels=by_labels, + instance_selector=instance_selector, + use_shared_pool=use_shared_pool, + ) + + +def expr_sum_delta( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + range_selector: str = "$__rate_interval", + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the sum of delta of a metric. + + Example: + + sum(delta( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (instance) + """ + return expr_aggr_func( + metric=metric, + aggr_op="sum", + func="delta", + label_selectors=label_selectors, + range_selector=range_selector, + by_labels=by_labels, + instance_selector=instance_selector, + ) + + +def expr_sum_increase( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + range_selector: str = "$__rate_interval", + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the sum of increase of a metric. + + Example: + + sum(increase( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (instance) + """ + return expr_aggr_func( + metric=metric, + aggr_op="sum", + func="increase", + instance_selector=instance_selector, + label_selectors=label_selectors, + range_selector=range_selector, + by_labels=by_labels, + ) + + +def expr_sum_aggr_over_time( + metric: str, + aggr: str, + range_selector: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the sum of average value of all points in the specified interval of a metric. + + Example: + + sum(avg_over_time( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [1m] + )) by (instance) + """ + return expr_aggr_func( + metric=metric, + aggr_op="sum", + func=f"{aggr}_over_time", + label_selectors=label_selectors, + range_selector=range_selector, + by_labels=by_labels, + instance_selector=instance_selector, + ) + + +def expr_max_rate( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the max of rate of a metric. + + Example: + + max(rate( + tikv_thread_voluntary_context_switches + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (name) + """ + # $__rate_interval is a Grafana variable that is specialized for Prometheus + # rate and increase function. + # See https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/ + return expr_aggr_func( + metric=metric, + aggr_op="max", + func="rate", + label_selectors=label_selectors, + range_selector="$__rate_interval", + by_labels=by_labels, + instance_selector=instance_selector, + ) + + +def expr_count_rate( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> Expr: + """ + Calculate the count of rate of a metric. + + Example: + + count(rate( + tikv_thread_cpu_seconds_total + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",name=~"sst_.*"} + [$__rate_interval] + )) by (instance) + """ + # $__rate_interval is a Grafana variable that is specialized for Prometheus + # rate and increase function. + # See https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/ + return expr_aggr_func( + metric=metric, + aggr_op="count", + func="rate", + label_selectors=label_selectors, + range_selector="$__rate_interval", + by_labels=by_labels, + instance_selector=instance_selector, + ) + + +def expr_simple( + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], +) -> Expr: + """ + Query an instant vector of a metric. + + Example: + + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + """ + expr = Expr(metric=metric) + expr.function("", label_selectors=label_selectors) + expr.instance_selector = tuple(instance_selector) + return expr + + +def expr_operator( + lhs: Union[Expr, OpExpr, str], operator: str, rhs: Union[Expr, OpExpr, str] +) -> OpExpr: + return OpExpr(lhs, operator, rhs) + + +def expr_histogram_quantile( + quantile: Union[float, str], + metrics: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = [], + is_optional_quantile: bool = False, + use_shared_pool: bool = False, +) -> Expr: + """ + Query a quantile of a histogram metric. + + Example: + + histogram_quantile(0.99, sum(rate( + tikv_grpc_msg_duration_seconds_bucket + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance",type!="kv_gc"} + [$__rate_interval] + )) by (le)) + """ + # sum(rate(metrics_bucket{label_selectors}[$__rate_interval])) by (le) + assert not metrics.endswith( + "_bucket" + ), f"'{metrics}' should not specify '_bucket' suffix manually" + by_labels = list(filter(lambda label: label != "le", by_labels)) + sum_rate_of_buckets = expr_sum_rate( + metrics + "_bucket", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels + ["le"], + use_shared_pool=use_shared_pool, + ) + # histogram_quantile({quantile}, {sum_rate_of_buckets}) + # Keep string quantiles as-is so "0.80" / "1.00" match legacy PromQL. + quantile_str = quantile if isinstance(quantile, str) else f"{quantile}" + if is_optional_quantile: + quantile_str = OPTIONAL_QUANTILE_INPUT + expr = expr_aggr( + metric=sum_rate_of_buckets, + aggr_op="histogram_quantile", + aggr_param=quantile_str, + instance_selector=(), + label_selectors=[], + by_labels=[], + ).extra( + # Do not attach default label selector again. + default_label_selectors=[], + ) + # Do not set use_shared_pool for the outer expression. It only needs to be set to the innermost layer. + return expr + + +def expr_topk( + k: int, + metrics: Union[str, Expr], + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, +) -> Expr: + """ + Query topk of a metric. + + Example: + + topk(20, tikv_thread_voluntary_context_switches) + """ + # topk({k}, {metric}) — outer clears cluster/instance defaults (metric may already be an Expr). + inner = ( + expr_simple(metrics, instance_selector=instance_selector) + if isinstance(metrics, str) + else metrics + ) + expr = expr_aggr( + metric=inner, + aggr_op="topk", + aggr_param=f"{k}", + instance_selector=(), + label_selectors=[], + by_labels=[], + ).extra( + # Do not attach default label selector again. + default_label_selectors=[] + ) + return expr + + +def expr_histogram_avg( + metrics: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = ["instance"], +) -> OpExpr: + """ + Query the avg of a histogram metric. + + Example: + + sum(rate( + tikv_grpc_msg_duration_seconds_sum + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance"} + [$__rate_interval] + )) / sum(rate( + tikv_grpc_msg_duration_seconds_count + {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster",instance=~"$instance"} + [$__rate_interval] + )) + """ + for suffix in ["_bucket", "_count", "_sum"]: + assert not metrics.endswith( + suffix + ), f"'{metrics}' should not specify '{suffix}' suffix manually" + + return OpExpr( + expr_sum_rate( + metrics + "_sum", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + "/", + expr_sum_rate( + metrics + "_count", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + ) + + +def target( + expr: Union[Expr, OpExpr, str], + legend_format: Optional[str] = None, + hide=False, + data_source=DATASOURCE, + interval_factor=2, # Align with legacy TiFlash jsonnet helpers + # It indicates whether to add additional groupby label to the groupby and legend. + # If the expr is already groupby by instance, no need to set it to True. + # Additional groupby is used to support optional by instance. By default, it doesn't + # take effect. When the variable $additional_groupby of dashboard is set to `instance`. + # The expr would be groupby by `instance`. + additional_groupby=False, +) -> Target: + if isinstance(expr, Expr): + if legend_format is None and expr.by_labels: + legend_format = "-".join( + map(lambda x: "{{" + f"{x}" + "}}", expr.by_labels) + ) + if additional_groupby: + expr.append_by_labels(ADDITIONAL_GROUPBY) + if legend_format is None: + legend_format = "" + legend_format += " {{" + ADDITIONAL_GROUPBY + "}}" + elif isinstance(expr, OpExpr): + assert legend_format is not None, "legend_format must be specified" + if additional_groupby: + expr.append_by_labels(ADDITIONAL_GROUPBY) + legend_format += " {{" + ADDITIONAL_GROUPBY + "}}" + + return Target( + expr=f"{expr}", + hide=hide, + legendFormat=legend_format, + intervalFactor=interval_factor, + datasource=data_source, + ) + + +def template( + name, + type, + query, + data_source, + hide, + regex=None, + multi=False, + include_all=False, + all_value=None, + label=None, + refresh=2, +) -> Template: + return Template( + dataSource=data_source, + hide=hide, + label=name if label is None else label, + multi=multi, + name=name, + query=query, + refresh=refresh, + sort=1, + type=type, + useTags=False, + regex=regex, + includeAll=include_all, + allValue=all_value, + ) + + +class Layout: + # Rows are always 24 "units" wide. + ROW_WIDTH = 24 + PANEL_HEIGHT = 8 + row_panel: RowPanel + current_row_y_pos: int + current_row_x_pos: int + + def __init__(self, title, collapsed=True, repeat: Optional[str] = None) -> None: + extraJson = None + if repeat: + extraJson = {"repeat": repeat} + title = f"{title} - ${repeat}" + self.current_row_y_pos = 0 + self.current_row_x_pos = 0 + self.row_panel = RowPanel( + title=title, + gridPos=GridPos(h=self.PANEL_HEIGHT, w=self.ROW_WIDTH, x=0, y=0), + collapsed=collapsed, + extraJson=extraJson, + ) + + def row( + self, + panels: list[Panel], + width: int = ROW_WIDTH, + height: Optional[int] = None, + widths: Optional[list[int]] = None, + ): + """Start a new band and pack panels left-to-right. + + - width: total width budget for this band (default 24) + - height: band height (default PANEL_HEIGHT) + - widths: optional per-panel widths; when omitted, split evenly + """ + count = len(panels) + if count == 0: + return panels + h = self.PANEL_HEIGHT if height is None else height + if widths is not None: + assert len(widths) == count + ws = list(widths) + else: + base = width // count + ws = [base] * count + ws[-1] += width - base * count + x = 0 # each band starts at the left edge + for panel, w in zip(panels, ws): + panel.gridPos = GridPos(h=h, w=w, x=x, y=self.current_row_y_pos) + x += w + self.row_panel.panels.extend(panels) + self.current_row_y_pos += h + # Do not carry x into the next band; authors call row()/half_row() per band. + self.current_row_x_pos = 0 + + def half_row(self, panels: list[Panel], height: Optional[int] = None): + """Place panels in the left half (width 12), leaving the right half empty when alone.""" + self.row(panels, self.ROW_WIDTH // 2, height=height) + + +def timeseries_panel( + title, + targets, + legend_calcs=["max", "last"], + unit="s", + draw_style="line", + line_width=1, + fill_opacity=10, + gradient_mode="opacity", + tooltip_mode="multi", + legend_display_mode="table", + legend_placement="right", + description=None, + data_source=DATASOURCE, +) -> TimeSeries: + return TimeSeries( + title=title, + dataSource=data_source, + description=description, + targets=targets, + legendCalcs=legend_calcs, + drawStyle=draw_style, + lineWidth=line_width, + fillOpacity=fill_opacity, + gradientMode=gradient_mode, + unit=unit, + tooltipMode=tooltip_mode, + legendDisplayMode=legend_display_mode, + legendPlacement=legend_placement, + ) + + +def yaxis( + format: str, + log_base: int = 1, + min=None, + max=None, + label: Optional[str] = None, + decimals: Optional[int] = None, + show: bool = True, +) -> YAxis: + # Prefer IEC byte units (Grafana "bytes"/"binBps"); forbid SI ("decbytes"/"Bps"). + assert format not in [ + UNITS.BYTES, + UNITS.BITS, + UNITS.KILO_BYTES, + UNITS.MEGA_BYTES, + UNITS.GIGA_BYTES, + UNITS.TERA_BYTES, + UNITS.PETA_BYTES, + UNITS.BYTES_SEC, + UNITS.KILO_BYTES_SEC, + UNITS.MEGA_BYTES_SEC, + UNITS.GIGA_BYTES_SEC, + UNITS.TERA_BYTES_SEC, + UNITS.PETA_BYTES_SEC, + ], "Must not use SI bytes" + return YAxis( + format=format, + logBase=log_base, + min=min, + max=max, + label=label, + decimals=decimals, + show=show, + ) + + +def yaxes( + left_format: str, + right_format: Optional[str] = None, + log_base: int = 1, + left_min=None, + left_max=None, + left_label: Optional[str] = None, + left_decimals: Optional[int] = None, + left_log_base: Optional[int] = None, + left_show: bool = True, + right_min=None, + right_max=None, + right_label: Optional[str] = None, + right_decimals: Optional[int] = None, + right_log_base: Optional[int] = None, + right_show: bool = False, +) -> YAxes: + """Build Y axes. Right axis is hidden by default; pass right_show=True for dual-axis panels.""" + left = yaxis( + left_format, + log_base=log_base if left_log_base is None else left_log_base, + min=left_min, + max=left_max, + label=left_label, + decimals=left_decimals, + show=left_show, + ) + # Always set right so grafanalib's default (show=True) never leaks into single-axis panels. + right = yaxis( + right_format if right_format is not None else UNITS.SHORT, + log_base=log_base if right_log_base is None else right_log_base, + min=right_min, + max=right_max, + label=right_label, + decimals=right_decimals, + show=right_show, + ) + return YAxes(left=left, right=right) + + +def graph_legend( + avg=False, + current=True, + max=True, + min=False, + show=True, + total=False, + align_as_table=True, + hide_empty=True, + hide_zero=True, + right_side=True, + side_width=None, + sort_desc=True, +) -> Legend: + sort = "max" if max else "current" + return Legend( + avg=avg, + current=current, + max=max, + min=min, + show=show, + total=total, + alignAsTable=align_as_table, + hideEmpty=hide_empty, + hideZero=hide_zero, + rightSide=right_side, + sideWidth=side_width, + sort=sort, + sortDesc=sort_desc, + ) + + +def graph_panel( + title: str, + targets: list[Target], + description=None, + yaxes=yaxes(left_format=UNITS.NONE_FORMAT), + legend=None, + tooltip=Tooltip(shared=True, valueType="individual"), + lines=True, + line_width=1, + fill=0, + fill_gradient=0, + stack=False, + thresholds: list[GraphThreshold] = [], + series_overrides: list = [], + data_source=DATASOURCE, + null_point_mode=NULL_AS_ZERO, + tooltip_sort: int = 0, + decimals: Optional[int] = None, + points: bool = False, + pointradius: int = 5, +) -> Panel: + # Internal extraJson patches grafanalib gaps (fillGradient / tooltip.sort / decimals). + extraJson: dict = {} + if fill_gradient != 0: + # fillGradient is only valid when fill is 1. + if fill == 0: + fill = 1 + # fillGradient is not set correctly in grafanalib(0.7.0), so we need to + # set it manually. + # TODO: remove it when grafanalib fix this. + extraJson["fillGradient"] = 1 + + if tooltip_sort: + extraJson["tooltip"] = { + "shared": True, + "sort": tooltip_sort, + "value_type": getattr(tooltip, "valueType", "individual"), + } + for target in targets: + # Make sure target is in time_series format. + if getattr(target, "format", None) in (None, TIME_SERIES_TARGET_FORMAT): + target.format = TIME_SERIES_TARGET_FORMAT + + if decimals is not None: + extraJson["decimals"] = decimals + + return Graph( + title=title, + dataSource=data_source, + description=description, + targets=targets, + yAxes=yaxes, + legend=legend if legend else graph_legend(), + lines=lines, + bars=not lines, + lineWidth=line_width, + fill=fill, + fillGradient=fill_gradient, + stack=stack, + nullPointMode=null_point_mode, + thresholds=thresholds, + tooltip=tooltip, + seriesOverrides=series_overrides, + # Do not specify max max data points, let Grafana decide. + maxDataPoints=None, + points=points, + pointRadius=pointradius, + extraJson=extraJson or None, + ) + + +def series_override( + alias: str, + bars: bool = False, + lines: bool = True, + yaxis: int = 1, + fill: int = 1, + zindex: int = 0, + dashes: Optional[bool] = None, + dash_length: Optional[int] = None, + space_length: Optional[int] = None, + transform_negative_y: bool = False, +) -> SeriesOverride: + class SeriesOverridePatch(SeriesOverride): + dashes_override: Optional[bool] + dash_length_override: Optional[int] + space_length_override: Optional[int] + transform_negative_y: bool + + def __init__(self, *args, **kwargs) -> None: + self.dashes_override = kwargs["dashes"] + if self.dashes_override is None: + del kwargs["dashes"] + self.dash_length_override = kwargs["dashLength"] + if self.dash_length_override is None: + del kwargs["dashLength"] + self.space_length_override = kwargs["spaceLength"] + if self.space_length_override is None: + del kwargs["spaceLength"] + self.transform_negative_y = kwargs["transform_negative_y"] + del kwargs["transform_negative_y"] + super().__init__(*args, **kwargs) + + def to_json_data(self): + data = super().to_json_data() + # The default 'null' color makes it transparent, remove it. + del data["color"] + # The default 'null' makes it a transparent line, remove it. + if self.dashes_override is None: + del data["dashes"] + if self.dash_length_override is None: + del data["dashLength"] + if self.space_length_override is None: + del data["spaceLength"] + # Add missing transform. + if self.transform_negative_y: + data["transform"] = "negative-Y" + return data + + return SeriesOverridePatch( + alias=alias, + bars=bars, + lines=lines, + yaxis=yaxis, + fill=fill, + zindex=zindex, + dashes=dashes, + dashLength=dash_length, + spaceLength=space_length, + transform_negative_y=transform_negative_y, + ) + + +def heatmap_color() -> HeatmapColor: + return HeatmapColor( + cardColor="#b4ff00", + colorScale="sqrt", + colorScheme="interpolateSpectral", + exponent=0.5, + mode="spectrum", + max=None, + min=None, + ) + + +def heatmap_panel( + title: str, + metric: str, + description=None, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + y_axis=None, + tooltip=Tooltip(shared=True, valueType="individual"), + color=heatmap_color(), + data_source=DATASOURCE, + # TiFlash Summary defaults to sum(delta); pass func="increase" for CSE-style. + func: str = "delta", +) -> Panel: + assert metric.endswith( + "_bucket" + ), f"'{metric}' should be a histogram metric with '_bucket' suffix" + # Build a fresh YAxis per call. A shared default object would be mutated across + # panels (e.g. via decimals), so later heatmaps would inherit earlier settings. + if y_axis is None: + y_axis = yaxis(UNITS.NO_FORMAT, decimals=1) + expr_fn = expr_sum_increase if func == "increase" else expr_sum_delta + t = target( + expr=expr_fn( + metric, + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=["le"], + ), + ) + # Make sure targets are in heatmap format. + t.format = "heatmap" + # Heatmap target legendFormat should be "{{le}}" + t.legendFormat = "{{le}}" + return Heatmap( + title=title, + dataSource=data_source, + description=description, + targets=[t], + yAxis=y_axis, + color=color, + dataFormat="tsbuckets", + yBucketBound="upper", + tooltip=tooltip, + extraJson={"tooltip": {"showHistogram": True, "show": True}}, + hideZeroBuckets=True, + # Limit data points, because too many data points slows browser when + # the resolution is too high. + # See: https://grafana.com/blog/2020/06/23/how-to-visualize-prometheus-histograms-in-grafana/ + maxDataPoints=512, + # Fix grafana heatmap migration panic if options is null. + # See: https://github.com/grafana/grafana/blob/v9.5.14/public/app/plugins/panel/heatmap/migrations.ts#L17 + options={}, + ) + + +def stat_panel( + title: str, + targets: list[Target], + description=None, + format=UNITS.NONE_FORMAT, + graph_mode="none", + decimals: Optional[int] = None, + mappings: Optional[StatValueMappings] = None, + text_mode: str = "auto", + data_source=DATASOURCE, +) -> Panel: + for target in targets: + # Make sure target is in time_series format. + target.format = TIME_SERIES_TARGET_FORMAT + return Stat( + title=title, + dataSource=data_source, + description=description, + targets=targets, + format=format, + graphMode=graph_mode, + reduceCalc="lastNotNull", + decimals=decimals, + mappings=mappings, + textMode=text_mode, + ) + + +def graph_panel_histogram_quantiles( + title: str, + description: str, + yaxes: YAxes, + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + by_labels: list[str] = [], + hide_p9999=False, + hide_avg=False, + hide_count=False, + additional_groupby=True, + legend_label_prefix="", +) -> Panel: + """ + Return a graph panel that shows histogram quantiles of a metric. + + Targets: + - 99.99% quantile + - 99% quantile + - avg + - count + """ + + def legend(prefix, labels): + if not labels: + return prefix + else: + return "-".join( + [prefix] + [f"{legend_label_prefix}{{{{{lb}}}}}" for lb in labels] + ) + + return graph_panel( + title=title, + description=description, + targets=[ + target( + expr=expr_histogram_quantile( + 0.9999, + f"{metric}", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=legend("99.99%", by_labels), + hide=hide_p9999, + additional_groupby=additional_groupby, + ), + target( + expr=expr_histogram_quantile( + 0.99, + f"{metric}", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=legend("99%", by_labels), + additional_groupby=additional_groupby, + ), + target( + expr=expr_histogram_avg( + metric, + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=legend("avg", by_labels), + hide=hide_avg, + additional_groupby=additional_groupby, + ), + target( + expr=expr_sum_rate( + f"{metric}_count", + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=legend("count", by_labels), + hide=hide_count, + additional_groupby=additional_groupby, + ), + ], + yaxes=yaxes, + series_overrides=[ + series_override( + # use regex because the real alias is "count ${additional_groupby}" + alias="/^count/", + fill=2, + yaxis=2, + zindex=-3, + dashes=True, + dash_length=1, + space_length=1, + transform_negative_y=True, + ), + series_override( + # use regex because the real alias is "avg ${additional_groupby}" + alias="/^avg/", + fill=7, + ), + ], + ) + + +def heatmap_panel_graph_panel_histogram_quantile_pairs( + heatmap_title: str, + heatmap_description: str, + graph_title: str, + graph_description: str, + yaxis_format: str, + metric: str, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors=[], + graph_by_labels=[], + graph_hides: list[str] = ["count"], +) -> list[Panel]: + hide_count = False + hide_avg = False + for hide in graph_hides: + if hide == "count": + hide_count = True + elif hide == "avg": + hide_avg = True + + return [ + heatmap_panel( + title=heatmap_title, + description=heatmap_description, + y_axis=yaxis(format=yaxis_format, decimals=1), + metric=f"{metric}_bucket", + instance_selector=instance_selector, + label_selectors=label_selectors, + ), + graph_panel_histogram_quantiles( + title=graph_title, + description=graph_description, + metric=f"{metric}", + yaxes=yaxes(left_format=yaxis_format), + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=graph_by_labels, + hide_count=hide_count, + hide_avg=hide_avg, + ), + ] + + +# --------------------------------------------------------------------------- +# TiFlash Summary L3 helpers (ported from tiflashnet/common.libsonnet) +# --------------------------------------------------------------------------- + +S3_STYLE_QUANTILES = [ + {"q": "1.00", "name": "max", "hide": True, "max_hack": True}, + {"q": "0.9999", "name": "9999"}, + {"q": "0.999", "name": "999", "hide": True}, + {"q": "0.99", "name": "99"}, + {"q": "0.80", "name": "80", "hide": True}, + {"name": "avg", "hide": True, "avg": True}, +] + + +def _strip_bucket(metric: str) -> str: + return metric[: -len("_bucket")] if metric.endswith("_bucket") else metric + + +def by_legend(by_labels: list[str]) -> str: + return "-".join("{{" + l + "}}" for l in by_labels) + + +def tiflash_override( + alias: str, + yaxis: Optional[int] = None, + color: Optional[str] = None, + linewidth: Optional[int] = None, + fill: Optional[int] = None, + hide_tooltip: Optional[bool] = None, + legend: Optional[bool] = None, + null_point_mode: Optional[str] = None, + dashes: Optional[bool] = None, + zindex: Optional[int] = None, +) -> dict: + """Series override as plain dict (grafanalib SeriesOverride is limited).""" + data: dict = {"alias": alias} + if yaxis is not None: + data["yaxis"] = yaxis + if color is not None: + data["color"] = color + if linewidth is not None: + data["linewidth"] = linewidth + if fill is not None: + data["fill"] = fill + if hide_tooltip is not None: + data["hideTooltip"] = hide_tooltip + if legend is not None: + data["legend"] = legend + if null_point_mode is not None: + data["nullPointMode"] = null_point_mode + if dashes is not None: + data["dashes"] = dashes + if zindex is not None: + data["zindex"] = zindex + return data + + +def duration_panel( + title: str, + metric: str, + description=None, + by_labels: list[str] = [], + legend: str = "%s {{$additional_groupby}}", + unit: str = UNITS.SECONDS, + y_right: str = UNITS.SHORT, + quantiles=None, + show_avg: Optional[bool] = None, + extra_targets: list[Target] = [], + series_overrides: list = [], + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + extra_label_selectors: list[str] = [], +) -> Panel: + """Duration histogram panel with S3-style quantile visibility defaults.""" + metric_base = _strip_bucket(metric) + qs = [dict(q) for q in (quantiles if quantiles is not None else S3_STYLE_QUANTILES)] + if show_avg is not None: + for q in qs: + if q.get("avg"): + q["hide"] = not show_avg + + targets: list[Target] = [] + for q in qs: + hide = bool(q.get("hide")) + name = q["name"] + if q.get("avg"): + expr: Union[Expr, OpExpr, str] = expr_histogram_avg( + metric_base, + instance_selector=instance_selector, + label_selectors=extra_label_selectors, + by_labels=by_labels + [ADDITIONAL_GROUPBY], + ) + else: + # Match jsonnet by (le, , $additional_groupby) + bucket_rate = expr_sum_rate( + metric_base + "_bucket", + instance_selector=instance_selector, + label_selectors=extra_label_selectors, + by_labels=["le"] + by_labels + [ADDITIONAL_GROUPBY], + ) + q_str = str(q["q"]) + if q.get("max_hack"): + # maxHack: histogram_quantile(q, sum(round(1e9*rate(...)))/1e9) + # Keep as raw PromQL for exact parity with jsonnet. + sel = ( + list(CLUSTER_LABEL_SELECTORS) + + list(instance_selector) + + list(extra_label_selectors) + ) + sel_str = ",".join(sel) + by_clause = ", ".join(["le"] + by_labels + [ADDITIONAL_GROUPBY]) + expr = ( + f"histogram_quantile({q_str}, " + f"sum(round(1000000000*rate({metric_base}_bucket{{{sel_str}}}" + f"[$__rate_interval]))) by ({by_clause}) / 1000000000)" + ) + else: + expr = expr_aggr( + metric=bucket_rate, + aggr_op="histogram_quantile", + aggr_param=q_str, + instance_selector=(), + by_labels=[], + ).extra(default_label_selectors=[]) + targets.append(target(expr=expr, legend_format=legend % name, hide=hide)) + targets.extend(extra_targets) + return graph_panel( + title=title, + description=description, + targets=targets, + yaxes=yaxes( + left_format=unit, + right_format=y_right, + left_min="0", + right_show=bool(series_overrides), + ), + legend=graph_legend(max=True, current=True, sort_desc=True), + fill=1, + series_overrides=series_overrides, + tooltip_sort=2, + null_point_mode=NULL_AS_ZERO, + ) + + +def ops_panel( + title: str, + metric: str, + by_labels: list[str] = [], + legend: Optional[str] = None, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + description=None, + y_left: str = "ops", + y_right: Optional[str] = None, + fill: int = 0, + left_min="0", + left_decimals: Optional[int] = None, + decimals: Optional[int] = None, + null_point_mode=NULL_AS_ZERO, + panel_legend=None, +) -> Panel: + """Single sum(rate(...)) OPS-style panel. + + ``legend`` is the PromQL legendFormat string; pass ``panel_legend`` for + Grafana legend table options (``graph_legend(...)``). + """ + if legend is None: + legend = "value" if not by_labels else by_legend(by_labels) + t = target( + expr=expr_sum_rate( + metric, + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=legend, + ) + return graph_panel( + title=title, + description=description, + targets=[t], + yaxes=yaxes( + left_format=y_left, + right_format=y_right, + left_min=left_min, + left_decimals=left_decimals, + ), + legend=panel_legend, + fill=fill, + tooltip_sort=2, + null_point_mode=null_point_mode, + decimals=decimals, + ) + + +def tiflash_heatmap_panel( + title: str, + metric: str, + description=None, + y_format: str = UNITS.SECONDS, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + func: str = "delta", +) -> Panel: + return heatmap_panel( + title=title, + metric=metric, + description=description, + instance_selector=instance_selector, + label_selectors=label_selectors, + y_axis=yaxis(format=y_format, decimals=1), + func=func, + ) + + +def cpu_with_limit_panel( + title: str, + name_regex: str, + description=None, + legend: str = "{{instance}}", + metric: str = "tiflash_proxy_thread_cpu_seconds_total", + hide_limit: bool = False, + instance_selector: Sequence[str] = PROXY_LABEL_SELECTORS, +) -> Panel: + """Thread CPU + Limit count line (proxy instance + role selectors). + + When hide_limit=True, the Limit target is still included but hidden by default. + """ + targets = [ + target( + expr=expr_sum_rate( + metric, + instance_selector=instance_selector, + label_selectors=[f'name=~"{name_regex}"'], + by_labels=["instance"], + ), + legend_format=legend, + ), + target( + expr=expr_aggr( + metric, + "count", + instance_selector=instance_selector, + label_selectors=[f'name=~"{name_regex}"'], + by_labels=["instance"], + ), + legend_format="Limit", + hide=hide_limit, + ), + ] + overrides = [ + tiflash_override( + "Limit", + color="#F2495C", + hide_tooltip=True, + legend=False, + linewidth=2, + null_point_mode="connected", + ) + ] + return graph_panel( + title=title, + description=description, + targets=targets, + yaxes=yaxes( + left_format=UNITS.PERCENT_UNIT, + right_format=UNITS.SHORT, + left_min="0", + ), + fill=0, + fill_gradient=0, + series_overrides=overrides, + null_point_mode="null", + ) + + +def ops_hit_ratio_panel( + title: str, + metric: str, + ratios: list[dict], + by_labels: list[str] = [], + legend: Optional[str] = None, + instance_selector: Sequence[str] = CPP_LABEL_SELECTORS, + label_selectors: list[str] = [], + description=None, + y_left: str = "ops", + fill: int = 0, +) -> Panel: + ops_legend = ( + legend + if legend is not None + else ("ops" if not by_labels else by_legend(by_labels)) + ) + ops_t = target( + expr=expr_sum_rate( + metric, + instance_selector=instance_selector, + label_selectors=label_selectors, + by_labels=by_labels, + ), + legend_format=ops_legend, + ) + ratio_targets = [] + overrides = [] + for r in ratios: + r_metric = r.get("metric", metric) + r_by = r.get("by", []) + hit = r.get("hit_labels", r.get("hitLabels", [])) + total = r.get("total_labels", r.get("totalLabels", [])) + if isinstance(hit, str): + hit = [hit] if hit else [] + if isinstance(total, str): + total = [total] if total else [] + hit_expr = expr_sum_rate( + r_metric, + instance_selector=instance_selector, + label_selectors=hit, + by_labels=r_by, + ) + tot_expr = expr_sum_rate( + r_metric, + instance_selector=instance_selector, + label_selectors=total, + by_labels=r_by, + ) + ratio_targets.append( + target( + expr=expr_operator(hit_expr, "/", tot_expr), + legend_format=r["legend"], + hide=bool(r.get("hide")), + ) + ) + overrides.append( + tiflash_override( + r.get("override_alias", r.get("overrideAlias", r["legend"])), + yaxis=2, + ) + ) + return graph_panel( + title=title, + description=description, + targets=[ops_t] + ratio_targets, + yaxes=yaxes( + left_format=y_left, + right_format=UNITS.PERCENT_UNIT, + right_show=True, + ), + fill=fill, + fill_gradient=0, + series_overrides=overrides, + null_point_mode=NULL_AS_ZERO, + ) + + +def make_heatmap( + title: str, + targets: list[Target], + description=None, + y_format: str = UNITS.SECONDS, + log_base: int = 1, + hide_zero_buckets: bool = True, + max_data_points: int = 512, + data_source=DATASOURCE, +) -> Panel: + """Build a legacy heatmap panel from pre-built targets (PromQL preserved).""" + for t in targets: + t.format = "heatmap" + if not t.legendFormat: + t.legendFormat = "{{le}}" + yax = yaxis(format=y_format, log_base=log_base) + return Heatmap( + title=title, + dataSource=data_source, + description=description, + targets=targets, + yAxis=yax, + color=heatmap_color(), + dataFormat="tsbuckets", + yBucketBound="upper", + tooltip=Tooltip(shared=True, valueType="individual"), + extraJson={"tooltip": {"showHistogram": True, "show": True}}, + hideZeroBuckets=hide_zero_buckets, + maxDataPoints=max_data_points, + options={}, + ) diff --git a/metrics/grafana/generate_dashboard.sh b/metrics/grafana/generate_dashboard.sh new file mode 100755 index 00000000000..6b5186468cf --- /dev/null +++ b/metrics/grafana/generate_dashboard.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Copyright 2026 PingCAP, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +cd "$(dirname "$0")" + +if ! command -v uv >/dev/null 2>&1; then + echo "uv is required. Install: https://docs.astral.sh/uv/" >&2 + exit 1 +fi + +uv sync + +.venv/bin/isort --profile black *.py +.venv/bin/black *.py + +.venv/bin/generate-dashboard \ + -o tiflash_summary.json \ + tiflash_summary.dashboard.py + +# Checksum path prefix matches CSE style (repo-root relative). +( + cd ../.. + sha256sum ./metrics/grafana/tiflash_summary.json > metrics/grafana/tiflash_summary.json.sha256 +) + +echo "Generated $(pwd)/tiflash_summary.json" diff --git a/metrics/grafana/pyproject.toml b/metrics/grafana/pyproject.toml new file mode 100644 index 00000000000..58783e57603 --- /dev/null +++ b/metrics/grafana/pyproject.toml @@ -0,0 +1,24 @@ +# Copyright 2026 PingCAP, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[project] +name = "tiflash-grafana-dashboards" +version = "0.1.0" +requires-python = ">=3.10" +dependencies = [ + "attrs", + "grafanalib==0.7.1", + "black==26.5.1", + "isort==8.0.1", +] diff --git a/metrics/grafana/tiflash_summary.dashboard.py b/metrics/grafana/tiflash_summary.dashboard.py new file mode 100644 index 00000000000..662d5352a74 --- /dev/null +++ b/metrics/grafana/tiflash_summary.dashboard.py @@ -0,0 +1,5536 @@ +# Copyright 2026 PingCAP, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import sys + +sys.path.append(os.path.dirname(__file__)) + +from grafanalib.core import ( + GRAPH_TOOLTIP_MODE_SHARED_CROSSHAIR, + HIDE_VARIABLE, + SHOW, + Dashboard, + RowPanel, + Template, + Templating, +) + +from common import ( + ADDITIONAL_GROUPBY, + DATASOURCE, + DATASOURCE_INPUT, + PROXY_LABEL_SELECTORS, + Layout, + cpu_with_limit_panel, + duration_panel, + expr_aggr, + expr_avg, + expr_histogram_avg, + expr_histogram_quantile, + expr_max, + expr_operator, + expr_simple, + expr_sum, + expr_sum_delta, + expr_sum_increase, + expr_sum_irate, + expr_sum_rate, + graph_legend, + graph_panel, + make_heatmap, + ops_panel, + target, + template, + yaxes, +) + + +def Templates() -> Templating: + return Templating( + list=[ + template( + name="k8s_cluster", + type="query", + query="label_values(tiflash_system_profile_event_Query, k8s_cluster)", + data_source=DATASOURCE, + hide=HIDE_VARIABLE, + multi=False, + include_all=False, + all_value=None, + label="K8s-cluster", + refresh=2, + ), + template( + name="tidb_cluster", + type="query", + query='label_values(tiflash_system_profile_event_Query{k8s_cluster="$k8s_cluster"}, tidb_cluster)', + data_source=DATASOURCE, + hide=HIDE_VARIABLE, + multi=False, + include_all=False, + all_value=None, + label="tidb_cluster", + refresh=2, + ), + template( + name="instance", + type="query", + query='label_values(tiflash_system_profile_event_Query{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster"}, instance)', + data_source=DATASOURCE, + hide=SHOW, + multi=True, + include_all=True, + all_value=None, + label="Instance", + refresh=1, + ), + template( + name="proxy_instance", + type="query", + query='label_values(tiflash_proxy_process_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster"}, instance)', + data_source=DATASOURCE, + hide=SHOW, + multi=True, + include_all=True, + all_value=None, + label="Proxy Instance", + refresh=1, + ), + # Custom vars: pass explicit options so text/value/current match clinic JSON. + # grafanalib's comma-split ignores "All : .*" label:value syntax. + Template( + name="additional_groupby", + type="custom", + query="none,instance", + dataSource=None, + hide=SHOW, + label="additional_groupby", + default="none", + options=[ + {"selected": True, "text": "none", "value": "none"}, + {"selected": False, "text": "instance", "value": "instance"}, + ], + ), + Template( + name="tiflash_role", + type="custom", + query="All : .*, Write : .*write-tiflash.*, Compute : .*compute-tiflash.*", + dataSource=None, + hide=SHOW, + label="Role", + default=".*", + options=[ + {"selected": True, "text": "All", "value": ".*"}, + { + "selected": False, + "text": "Write", + "value": ".*write-tiflash.*", + }, + { + "selected": False, + "text": "Compute", + "value": ".*compute-tiflash.*", + }, + ], + ), + ] + ) + + +def Server() -> RowPanel: + layout = Layout(title="Server") + layout.row( + [ + graph_panel( + title="Store size", + description="The storage size per TiFlash instance.\n(Not including some disk usage of TiFlash-Proxy by now)", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_StoreSizeUsed", + label_selectors=['type=~""'], + by_labels=["instance"], + ), + legend_format="{{instance}}-local", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_StoreSizeUsedRemote", + by_labels=["instance"], + ), + legend_format="{{instance}}-remote", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + fill=5, + line_width=0, + stack=True, + decimals=3, + ), + graph_panel( + title="Available size", + description="The available capacity size per TiFlash instance", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_StoreSizeAvailable", + by_labels=["instance"], + ), + legend_format="{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + fill=5, + line_width=0, + stack=True, + decimals=3, + ), + graph_panel( + title="Capacity size", + description="The capacity size per TiFlash instance", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_StoreSizeCapacity", + by_labels=["instance"], + ), + legend_format="{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + fill=5, + line_width=0, + stack=True, + decimals=3, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Uptime", + description="TiFlash uptime since last restart", + targets=[ + target( + expr=expr_simple("tiflash_system_asynchronous_metric_Uptime"), + legend_format="{{instance}}", + ), + ], + yaxes=yaxes(left_format="dtdurations"), + legend=graph_legend(max=False), + ), + graph_panel( + title="Region", + description="The number of Regions on each TiFlash instance", + targets=[ + target( + expr=expr_sum( + "tiflash_proxy_tikv_raftstore_region_count", + instance_selector=PROXY_LABEL_SELECTORS, + label_selectors=[ + 'type="region"', + ], + by_labels=["instance"], + ), + legend_format="{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_proxy_tikv_raftstore_hibernated_peer_state", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["instance", "state"], + ), + legend_format="{{instance}}-{{state}}", + hide=True, + ), + ], + yaxes=yaxes(left_format="short"), + null_point_mode="null", + decimals=0, + ), + ] + ) + layout.row( + [ + graph_panel( + title="CPU Usage", + description="TiFlash CPU usage calculated with process CPU running seconds.", + targets=[ + target( + expr='rate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", job=~".*tiflash", instance=~"$proxy_instance", instance=~"$tiflash_role"}[$__rate_interval])', + legend_format="{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_LogicalCPUCores", + by_labels=["instance"], + ), + legend_format="limit-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0", left_decimals=1), + null_point_mode="null", + series_overrides=[ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + } + ], + ), + graph_panel( + title="Memory", + description="The memory usage per TiFlash instance", + targets=[ + target( + expr=expr_sum( + "tiflash_proxy_process_resident_memory_bytes", + instance_selector=PROXY_LABEL_SELECTORS, + label_selectors=[ + 'job=~".*tiflash"', + ], + by_labels=["instance"], + ), + legend_format="{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_MemoryCapacity", + by_labels=["instance"], + ), + legend_format="limit-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_retained", + by_labels=[], + ), + legend_format="retained", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_mapped", + by_labels=[], + ), + legend_format="mapped", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_resident", + by_labels=[], + ), + legend_format="resident", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_allocated", + by_labels=[], + ), + legend_format="allocated", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_active", + by_labels=[], + ), + legend_format="active", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_metadata_thp", + by_labels=[], + ), + legend_format="metadata_thp", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_jemalloc_metadata", + by_labels=[], + ), + legend_format="metadata", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_mimalloc_current_rss", + label_selectors=['job=~".*tiflash"'], + by_labels=[], + ), + legend_format="mimalloc_rss", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_mimalloc_current_commit", + label_selectors=['job=~".*tiflash"'], + by_labels=[], + ), + legend_format="mimalloc_commit", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_mmap_alive", + label_selectors=['job=~".*tiflash"'], + by_labels=[], + ), + legend_format="mmap", + hide=True, + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + null_point_mode="null", + series_overrides=[ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + } + ], + ), + ] + ) + layout.row( + [ + graph_panel( + title="IO Throughput", + targets=[ + target( + expr=expr_sum_irate( + "tiflash_proxy_threads_io_bytes_total", + instance_selector=PROXY_LABEL_SELECTORS, + label_selectors=[ + 'job=~".*tiflash"', + ], + by_labels=["instance"], + ), + legend_format="{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0", left_decimals=0), + null_point_mode="null", + ), + graph_panel( + title="Remote Store Summary (Disagg arch)", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_s3_store_summary_bytes", + by_labels=["instance", "store_id", "type"], + ), + legend_format="store-{{store_id}}-{{type}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + ] + ) + return layout.row_panel + + +def ThreadsCPU() -> RowPanel: + layout = Layout(title="Threads CPU") + layout.row( + [ + cpu_with_limit_panel( + "SST Import Service", + "sst_importer.*", + description="Involved when importing data.", + hide_limit=True, + ), + cpu_with_limit_panel( + "SST Apply", + "apply_low_.*", + description="Involved when importing data.", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Region Task", + "region_task.*", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "Region Worker", + "region_worker.*", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Raft Store", + "raftstore_.*", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "Apply Worker", + "apply_.*", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Storage Background (Small Tasks)", + r"bg_\\d+", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "Storage Background (Large Tasks)", + r"bg_block_\\d+", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Manual Compaction", + "m_compact_pool", + description="Involved when manually compacting the data.", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "GRPC Async Server", + "async_poller.*", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "GRPC Async Client", + "GRPCComp.*", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "FAP builder", + "fap_builder.*", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Snapshot Sender", + "snap_sender.*", + legend="{{name}} {{instance}}", + ), + cpu_with_limit_panel( + "Segment Scheduler", + "segment_sched.*", + legend="{{name}} {{instance}}", + ), + ] + ) + layout.row( + [ + cpu_with_limit_panel( + "Local Index Pool", + "LocalIndexPool*", + legend="pool-{{instance}}", + ), + cpu_with_limit_panel( + "Segment Reader", + "SegmentReader.*", + legend="{{name}} {{instance}}", + ), + ] + ) + return layout.row_panel + + +def Threads() -> RowPanel: + layout = Layout(title="Threads") + layout.row( + [ + graph_panel( + title="Threads state", + targets=[ + target( + expr=expr_sum( + "tiflash_proxy_threads_state", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["instance", "state"], + ), + legend_format="{{instance}}-{{state}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_proxy_threads_state", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["instance"], + ), + legend_format="{{instance}}-total", + ), + ], + yaxes=yaxes(left_format="none"), + fill=1, + ), + graph_panel( + title="Threads IO", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_threads_io_bytes_total", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["name", "io", ADDITIONAL_GROUPBY], + ).extra(extra_expr="> 1024"), + legend_format="{{name}}-{{io}} {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps"), + fill=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Thread Voluntary Context Switches", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_thread_voluntary_context_switches", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["instance", "name"], + ).extra(extra_expr="> 200"), + legend_format="{{instance}} - {{name}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none"), + fill=1, + ), + graph_panel( + title="Thread Nonvoluntary Context Switches", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_thread_nonvoluntary_context_switches", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=["instance", "name"], + ).extra(extra_expr="> 50"), + legend_format="{{instance}} - {{name}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none"), + fill=1, + ), + ] + ) + return layout.row_panel + + +def Coprocessor() -> RowPanel: + layout = Layout(title="Coprocessor") + layout.row( + [ + ops_panel( + "Request QPS", + "tiflash_coprocessor_request_count", + by_labels=["type"], + y_left="none", + ), + ops_panel( + "Executor QPS", + "tiflash_coprocessor_executor_count", + by_labels=["type"], + y_left="none", + ), + ] + ) + layout.row( + [ + duration_panel( + "Request Duration", + "tiflash_coprocessor_request_duration_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ops_panel( + "Error QPS", + "tiflash_coprocessor_request_error", + by_labels=["reason"], + y_left="none", + ), + ] + ) + layout.row( + [ + duration_panel( + "Request Handle Duration", + "tiflash_coprocessor_request_handle_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ops_panel( + "Response Bytes/Seconds", + "tiflash_coprocessor_response_bytes", + by_labels=["type"], + y_left="bytes", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Cop task memory usage", + targets=[ + target( + expr=expr_histogram_quantile( + "0.999", + "tiflash_coprocessor_request_memory_usage", + by_labels=["type"], + ), + legend_format="999-{{type}}", + interval_factor=1, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_coprocessor_request_memory_usage", + by_labels=["type"], + ), + legend_format="99-{{type}}", + interval_factor=1, + ), + target( + expr=expr_histogram_quantile( + "0.95", + "tiflash_coprocessor_request_memory_usage", + by_labels=["type"], + ), + legend_format="95-{{type}}", + interval_factor=1, + ), + target( + expr=expr_histogram_quantile( + "0.80", + "tiflash_coprocessor_request_memory_usage", + by_labels=["type"], + ), + legend_format="80-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + ), + ops_panel( + "Exchange Bytes/Seconds", + "tiflash_exchange_data_bytes", + by_labels=["type"], + y_left="bytes", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Threads of Rpc", + targets=[ + target( + expr=expr_max( + "tiflash_thread_count", + label_selectors=['type=~"rpc.*"', 'type!~".*max"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + graph_panel( + title="Handling Request Number", + targets=[ + target( + expr=expr_sum( + "tiflash_coprocessor_handling_request_count", + by_labels=["type"], + ), + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Threads", + targets=[ + target( + expr=expr_max( + "tiflash_thread_count", + label_selectors=['type!~".*max"', 'type!~"rpc.*"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + graph_panel( + title="Max Threads of Rpc", + targets=[ + target( + expr=expr_max( + "tiflash_thread_count", + label_selectors=['type=~"rpc.*"', 'type=~".*max"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="MPP Query count", + description="The MPP query count in TiFlash", + targets=[ + target( + expr=expr_max( + "tiflash_mpp_task_manager", by_labels=["instance", "type"] + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + graph_panel( + title="Max Threads", + targets=[ + target( + expr=expr_max( + "tiflash_thread_count", + label_selectors=['type=~".*max"', 'type!~"rpc.*"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Time of the Longest Live MPP Task", + targets=[ + target( + expr=expr_simple("tiflash_mpp_task_monitor"), + legend_format="{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="s", left_min="0"), + ), + graph_panel( + title="Data size in send and receive queue", + targets=[ + target( + expr=expr_simple("tiflash_exchange_queueing_data_bytes"), + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + ), + ] + ) + layout.row( + [ + ops_panel( + "Network Transmission", + "tiflash_network_transmission_bytes", + by_labels=["type"], + y_left="bytes", + ), + graph_panel( + title="Establish calldata details", + description="The establish calldata details", + targets=[ + target( + expr=expr_max( + "tiflash_establish_calldata_count", + label_selectors=['type != "new_request_calldata"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + return layout.row_panel + + +def TaskScheduler() -> RowPanel: + layout = Layout(title="Task Scheduler") + layout.row( + [ + graph_panel( + title="Min TSO", + description="the min_tso of each instance", + targets=[ + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="min_tso"'], + by_labels=["instance", "resource_group"], + ), + legend_format="{{instance}}-{{resource_group}}", + ), + ], + yaxes=yaxes(left_format="none", left_label="TSO", left_show=False), + legend=graph_legend(max=False), + fill=1, + points=True, + pointradius=1, + null_point_mode="null", + ), + graph_panel( + title="Estimated Thread Usage and Limit", + description="estimated thread usage in min-tso scheduler, and the sort/hard limit of estimated thread in scheduler.", + targets=[ + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="thread_soft_limit"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="estimated_thread_usage"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="thread_hard_limit"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="global_estimated_thread_usage"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="group_entry_count"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + ), + ], + yaxes=yaxes(left_format="none", left_label="Threads", left_log_base=10), + legend=graph_legend(max=False), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Active and Waiting Queries Count", + description="the count of active/ waiting queries", + targets=[ + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="waiting_queries_count"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="active_queries_count"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + ], + yaxes=yaxes(left_format="none", left_label="Queries"), + ), + graph_panel( + title="Active and Waiting Tasks Count", + description="the count of active/ waiting tasks", + targets=[ + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="waiting_tasks_count"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="active_tasks_count"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{type}}-{{resource_group}}", + ), + ], + yaxes=yaxes(left_format="none", left_label="Tasks"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Hard Limit Exceeded Count", + description="the usage of estimated threads exceeded the hard limit where errors occur.", + targets=[ + target( + expr=expr_max( + "tiflash_task_scheduler", + label_selectors=['type="hard_limit_exceeded_count"'], + by_labels=["instance", "type", "resource_group"], + ), + legend_format="{{instance}}-{{resource_group}}", + ), + ], + yaxes=yaxes(left_format="none"), + ), + duration_panel( + "Task Waiting Duration", + "tiflash_task_scheduler_waiting_duration_seconds", + description="the time of waiting for schedule", + by_labels=["instance", "resource_group"], + legend="{{instance}}-{{resource_group}}-%s", + ), + ] + ) + return layout.row_panel + + +def DDL() -> RowPanel: + layout = Layout(title="DDL") + layout.row( + [ + graph_panel( + title="Schema Internal DDL OPM", + description="Executed DDL jobs per minute", + targets=[ + target( + expr='avg(increase(tiflash_schema_internal_ddl_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) by (type)', + legend_format="{{type}}", + interval_factor=1, + ), + target( + expr='sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))', + legend_format="total", + interval_factor=1, + ), + target( + expr=expr_sum_increase( + "tiflash_schema_internal_ddl_count", + by_labels=["type", "instance"], + ), + legend_format="{{type}}-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum_increase( + "tiflash_schema_internal_ddl_count", by_labels=["instance"] + ), + legend_format="total-{{instance}}", + hide=True, + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="opm", left_min="0"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + ), + graph_panel( + title="Schema Apply OPM", + description="Executed DDL apply jobs per minute", + targets=[ + target( + expr='avg(increase(tiflash_schema_trigger_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) by (type)', + legend_format="triggle-by-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="opm", left_min="0"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + ), + ] + ) + layout.half_row( + [ + graph_panel( + title="Schema Apply Duration", + targets=[ + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)', + legend_format="max-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.9999", + "tiflash_schema_apply_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="9999-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.999", + "tiflash_schema_apply_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="999-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_schema_apply_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="99-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.80", + "tiflash_schema_apply_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="80-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_avg( + "tiflash_schema_apply_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="avg-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_sum( + "tiflash_sync_schema_applying", + label_selectors=['type=~"$type"'], + by_labels=["instance"], + ), + legend_format="applying-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="s", right_format="short", left_min="0", right_show=True + ), + fill=1, + tooltip_sort=2, + series_overrides=[{"alias": "/^applying/", "yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def ImbalanceReadWrite() -> RowPanel: + layout = Layout(title="Imbalance read/write") + layout.row( + [ + graph_panel( + title="CPU Usage (irate)", + description="TiFlash CPU usage calculated with process CPU running seconds.", + targets=[ + target( + expr='irate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", job=~".*tiflash", instance=~"$tiflash_role"}[$__rate_interval])', + legend_format="{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_LogicalCPUCores", + by_labels=["instance"], + ), + legend_format="limit-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0", left_decimals=1), + legend=graph_legend(side_width=250), + null_point_mode="null", + series_overrides=[ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + } + ], + ), + graph_panel( + title="Segment Reader", + targets=[ + target( + expr='sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"SegmentReader.*", instance=~"$tiflash_role"}[$__rate_interval])) by (instance)', + legend_format="{{name}} {{instance}}", + ), + target( + expr='count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"SegmentReader.*", instance=~"$tiflash_role"}) by (instance)', + legend_format="Limit", + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0"), + null_point_mode="null", + series_overrides=[ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + "nullPointMode": "connected", + } + ], + ), + ] + ) + layout.row( + [ + ops_panel( + "Request QPS by instance", + "tiflash_coprocessor_request_count", + by_labels=["type", "instance"], + y_left="none", + ), + graph_panel( + title="Read Throughput by instance", + description="The flow of different kinds of read operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes", + by_labels=["instance"], + ), + legend_format="File Descriptor-{{instance}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMReadBytes", + by_labels=["instance"], + ), + legend_format="Page-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMBackgroundReadBytes", + by_labels=["instance"], + ), + legend_format="PageBackGround-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Write Command OPS By Instance", + description="The total count of different kinds of commands received", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMWriteBlock", + by_labels=["instance", "type"], + ), + legend_format="write block-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_increase( + "tiflash_storage_command_count", + by_labels=["instance", "type"], + ), + legend_format="{{type}}-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="opm", + left_min="0", + right_min="0", + right_show=True, + ), + series_overrides=[{"alias": "/delete_range|ingest/", "yaxis": 2}], + ), + graph_panel( + title="Write Throughput By Instance", + description="The throughput of write by instance", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"write"'], + by_labels=["instance"], + ), + legend_format="write-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"ingest"'], + by_labels=["instance"], + ), + legend_format="ingest-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="binBps", + right_format="bytes", + left_min="0", + right_show=True, + ), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + series_overrides=[{"alias": "/total/", "yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def MemoryTrace() -> RowPanel: + layout = Layout(title="Memory trace") + layout.row( + [ + graph_panel( + title="Number of Keyspaces", + targets=[ + target( + expr=expr_simple("tiflash_system_current_metric_NumKeyspace"), + legend_format="keyspace-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short"), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="Number of Physical Tables", + targets=[ + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_NumStorageDeltaMerge" + ), + legend_format="tables-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_simple("tiflash_system_current_metric_NumIStorage"), + legend_format="tables-all-{{instance}}", + hide=True, + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short"), + fill=1, + null_point_mode="null", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Number of Segments", + targets=[ + target( + expr=expr_simple("tiflash_system_current_metric_DT_NumSegment"), + legend_format="segments-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_NumMemTable" + ), + legend_format="mem_table-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short"), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="Bytes of MemTables", + targets=[ + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_BytesMemTable" + ), + legend_format="bytes-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_BytesMemTableAllocated" + ), + legend_format="bytes-allocated-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes"), + fill=1, + null_point_mode="null", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Mark Cache and Minmax Index Cache Memory Usage", + description="The memory usage of mark cache and minmax index cache", + targets=[ + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_MarkCacheBytes" + ), + legend_format="mark_cache_{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_MinMaxIndexFiles" + ), + legend_format="minmax_index_cache_{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_RNMVCCIndexCacheBytes" + ), + legend_format="rn_mvcc_index_cache_{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + series_overrides=[ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + } + ], + ), + graph_panel( + title="Effectiveness of Mark Cache", + description="cache misses or cache hits of mark_cache.\nBased on this infactor, we can check whether mark_cache is large enough", + targets=[ + target( + expr=expr_max( + "tiflash_system_profile_event_MarkCacheMisses", + by_labels=["instance"], + ), + legend_format="mark cache misses", + ), + target( + expr=expr_max( + "tiflash_system_profile_event_MarkCacheHits", + by_labels=["instance"], + ), + legend_format="mark cache hits", + ), + ], + yaxes=yaxes(left_format="percentunit"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + null_point_mode="null", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Schema of Column File", + description="Information about schema of column file, to learn the memory usage of schema", + targets=[ + target( + expr=expr_max( + "tiflash_shared_block_schemas", + label_selectors=['type=~"current_size"'], + by_labels=["instance"], + ), + legend_format="current_size-{{instance}}", + ), + target( + expr=expr_sum_rate( + "tiflash_shared_block_schemas", + label_selectors=['type=~"hit_count"'], + by_labels=["instance"], + ), + legend_format="hit_count_ops-{{instance}}", + ), + target( + expr=expr_max( + "tiflash_shared_block_schemas", + label_selectors=['type=~"still_used_when_evict"'], + by_labels=["instance"], + ), + legend_format="still_used_when_evict-{{instance}}", + ), + target( + expr=expr_sum_rate( + "tiflash_shared_block_schemas", + label_selectors=['type=~"miss_count"'], + by_labels=["instance"], + ), + legend_format="miss_count_ops-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short"), + legend=graph_legend(max=False), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="Read Snapshots", + targets=[ + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SegmentReadTasks" + ), + legend_format="read_tasks-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime" + ), + legend_format="max_snapshot_lifetime-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="short", + right_format="s", + left_min="0", + right_min="0", + right_show=True, + ), + fill=1, + null_point_mode="null", + series_overrides=[{"alias": "/max_snapshot_lifetime/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Memory by thread", + targets=[ + target( + expr='rate(tiflash_storages_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"alloc_.*"}[$__interval])', + legend_format="{{instance}}-{{type}}", + ), + target( + expr='-rate(tiflash_storages_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dealloc_.*"}[$__interval])', + legend_format="{{instance}}-{{type}}", + ), + target( + expr=expr_simple( + "tiflash_storages_thread_memory_usage", + label_selectors=['type=~"alloc_.*"'], + ), + legend_format="{{instance}}-{{type}}-tot", + hide=True, + ), + target( + expr='-tiflash_storages_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dealloc_.*"}', + legend_format="{{instance}}-{{type}}-tot", + hide=True, + ), + ], + yaxes=yaxes(left_format="bytes"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="Memory by thread (proxy)", + targets=[ + target( + expr='rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"alloc_.*"}[$__interval])', + legend_format="{{instance}}-{{type}}", + ), + target( + expr='-rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dealloc_.*"}[$__interval])', + legend_format="{{instance}}-{{type}}", + ), + target( + expr=expr_simple( + "tiflash_raft_proxy_thread_memory_usage", + label_selectors=['type=~"alloc_.*"'], + ), + legend_format="{{instance}}-{{type}}-tot", + hide=True, + ), + target( + expr='-tiflash_raft_proxy_thread_memory_usage{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dealloc_.*"}', + legend_format="{{instance}}-{{type}}-tot", + hide=True, + ), + ], + yaxes=yaxes(left_format="bytes"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + null_point_mode="null", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Memory by class", + targets=[ + target( + expr=expr_simple("tiflash_memory_usage_by_class"), + legend_format="{{instance}}-{{type}}", + ), + target( + expr='rate(tiflash_memory_usage_by_class{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__interval])', + legend_format="{{instance}}-{{type}}-rate", + hide=True, + ), + ], + yaxes=yaxes(left_format="bytes"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="KVStore memory", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_MemoryTrackingKVStore", + by_labels=["instance"], + ), + legend_format="{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + null_point_mode="null", + ), + ] + ) + return layout.row_panel + + +def ColumnarStorage() -> RowPanel: + layout = Layout(title="Columnar Storage") + layout.row( + [ + graph_panel( + title="IA usage", + targets=[ + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_ia_main_queue_capacity", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="capacity-main-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_ia_small_queue_capacity", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="capacity-small-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_ia_manager_segments_memory_capacity", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="capacity-segments-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_ia_manager_segments_memory_size", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="segments-mem-size-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_ia_manager_segments_disk_size", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="segments-disk-size-{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + series_overrides=[ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + } + ], + ), + duration_panel( + "IA Segments Memory Wait", + "tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds", + instance_selector=PROXY_LABEL_SELECTORS, + ), + ] + ) + layout.row( + [ + graph_panel( + title="IA Segment Remote Read Cache", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_ia_remote_read_segment_cache_hit", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="cache-hit {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_ia_remote_read_segment_cache_miss", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="cache-miss {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + duration_panel( + "IA Segments Remote Read Duration", + "tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds", + instance_selector=PROXY_LABEL_SELECTORS, + ), + ] + ) + layout.row( + [ + graph_panel( + title="ColumnarFile Cache", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_columnar_file_cache_hit", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="file-cache-hit {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_columnar_file_cache_miss", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="file-cache-miss {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + duration_panel( + "Columnar Prefetch Duration", + "tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds", + instance_selector=PROXY_LABEL_SELECTORS, + ), + duration_panel( + "Columnar Prefetch Cache Hit Duration", + "tiflash_proxy_kv_engine_columnar_prefetch_cache_hit", + instance_selector=PROXY_LABEL_SELECTORS, + ), + ] + ) + layout.row( + [ + ops_panel( + "Columnar Fetch Snapshot Retry", + "tiflash_proxy_kv_engine_columnar_fetch_snapshot_retry_count", + by_labels=[ADDITIONAL_GROUPBY], + legend="retry {{$additional_groupby}}", + instance_selector=PROXY_LABEL_SELECTORS, + ), + duration_panel( + "Columnar Fetch Snapshot Duration", + "tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds", + instance_selector=PROXY_LABEL_SELECTORS, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Columnar Meta Cache", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_columnar_meta_cache_hit", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="hit {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_columnar_meta_cache_miss", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="miss {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_proxy_kv_engine_columnar_meta_cache_parse", + instance_selector=PROXY_LABEL_SELECTORS, + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="parse {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + graph_panel( + title="Columnar Meta Cache Entries", + targets=[ + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_columnar_meta_cache_entries", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="entries-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + ), + graph_panel( + title="Columnar Meta Cache Weighted Size", + targets=[ + target( + expr=expr_simple( + "tiflash_proxy_kv_engine_columnar_meta_cache_weighted_size", + instance_selector=PROXY_LABEL_SELECTORS, + ), + legend_format="weighted_size-{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + ), + ] + ) + return layout.row_panel + + +def Storage() -> RowPanel: + layout = Layout(title="Storage") + layout.row( + [ + graph_panel( + title="Write Command OPS", + description="The total count of different kinds of commands received", + targets=[ + target( + expr=expr_sum_increase( + "tiflash_storage_command_count", by_labels=["type"] + ), + legend_format="{{type}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMWriteBlock", + by_labels=["type"], + ), + legend_format="write block", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="opm", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + series_overrides=[{"alias": "/delete_range|ingest/", "yaxis": 2}], + ), + graph_panel( + title="Write Amplification", + targets=[ + target( + expr='sum by (instance) ( tiflash_system_profile_event_PSMWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"} + tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"} ) / sum by (instance) ( tiflash_storage_throughput_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"write|ingest"} )', + legend_format="amp-total-{{instance}}", + interval_factor=1, + ), + target( + expr='sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[5m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"write|ingest"}[5m]) )', + legend_format="amp-5min-{{instance}}", + interval_factor=1, + ), + target( + expr='sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[10m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[10m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"write|ingest"}[10m]) )', + legend_format="amp-10min-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr='sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[30m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[30m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"write|ingest"}[30m]) )', + legend_format="amp-30min-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr='sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[5m]) )', + legend_format="fs-5min-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr='sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"write|ingest"}[5m]) )', + legend_format="write-5min-{{instance}}", + hide=True, + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="short", + right_format="binBps", + left_min="0", + left_max="20", + right_show=True, + ), + legend=graph_legend(max=False), + null_point_mode="null", + series_overrides=[{"alias": "/fs|write/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="SubTasks Write Throughput (bytes)", + description="The throughput of (maybe foreground) tasks of storage in bytes", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_subtask_throughput_bytes", + by_labels=["type"], + ), + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="binBps", + right_format="bytes", + left_min="0", + right_show=True, + ), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + series_overrides=[{"alias": "/total/", "yaxis": 2}], + ), + graph_panel( + title="SubTasks Write Throughput (rows)", + description="The throughput of (maybe foreground) tasks of storage in rows", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_subtask_throughput_rows", + by_labels=["type"], + ), + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="none", + right_format="bytes", + left_min="0", + right_show=True, + ), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + series_overrides=[{"alias": "/total/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + ops_panel( + "Small Internal Tasks OPS", + "tiflash_storage_subtask_count", + by_labels=["type"], + label_selectors=['type!~"(delta_merge|seg_merge|seg_split).*"'], + description="Total number of storage's internal sub tasks", + ), + duration_panel( + "Small Internal Tasks Duration", + "tiflash_storage_subtask_duration_seconds", + description="Duration of storage's internal sub tasks", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + extra_label_selectors=['type!~"(delta_merge|seg_merge|seg_split).*"'], + ), + ], + height=5, + ) + layout.row( + [ + ops_panel( + "Large Internal Tasks OPS", + "tiflash_storage_subtask_count", + by_labels=["type"], + label_selectors=['type=~"(delta_merge|seg_merge|seg_split).*"'], + description="Total number of storage's internal sub tasks", + ), + duration_panel( + "Large Internal Tasks Duration", + "tiflash_storage_subtask_duration_seconds", + description="Duration of storage's internal sub tasks", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + extra_label_selectors=['type=~"(delta_merge|seg_merge|seg_split).*"'], + ), + ], + height=5, + ) + layout.row( + [ + graph_panel( + title="Current Data Management Tasks", + description="The current processing number of segments' background management", + targets=[ + target( + expr=expr_avg( + "tiflash_system_current_metric_DT_DeltaMerge", + by_labels=["instance"], + ), + legend_format="delta_merge-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_DT_SegmentSplit", + by_labels=["instance"], + ), + legend_format="seg_split-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_DT_SegmentMerge", + by_labels=["instance"], + ), + legend_format="seg_merge-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short", left_min="0", left_decimals=0), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Opened File Count", + description="The number of currently opened file descriptors.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + targets=[ + target( + expr=expr_simple( + "tiflash_proxy_process_open_fds", + instance_selector=PROXY_LABEL_SELECTORS, + label_selectors=[ + 'job=~".*tiflash"', + ], + ), + legend_format="{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_OpenFileForWrite", + by_labels=["instance"], + ), + legend_format="W-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_OpenFileForRead", + by_labels=["instance"], + ), + legend_format="R-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_OpenFileForReadWrite", + by_labels=["instance"], + ), + legend_format="RW-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + ), + graph_panel( + title="File Open OPS", + description="The number of open file descriptors action.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_FileOpen", + by_labels=["instance"], + ), + legend_format="Open-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_FileOpenFailed", + by_labels=["instance"], + ), + legend_format="OpenFail-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + ), + graph_panel( + title="FSync Status", + description="OPS and duration of fsync operations.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_FileFSync", + by_labels=["instance"], + ), + legend_format="ops-fsync-{{instance}}", + interval_factor=1, + ), + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_system_seconds_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"fsync"}[$__rate_interval]))) by (le, instance) / 1000000000)', + legend_format="max-fsync-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="ops", right_format="s", left_min="0", right_show=True + ), + legend=graph_legend(side_width=250), + series_overrides=[{"alias": "/max-fsync/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Disk Write OPS", + description="The number of different kinds of read operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMWriteIOCalls", + by_labels=["type"], + ), + legend_format="Page", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMWritePages", by_labels=[] + ), + legend_format="PageFile", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_WriteBufferFromFileDescriptorWrite", + by_labels=[], + ), + legend_format="File Descriptor", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + graph_panel( + title="Disk Read OPS", + description="The number of different kinds of read operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMReadIOCalls", by_labels=[] + ), + legend_format="Page", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMReadPages", by_labels=[] + ), + legend_format="PageFile", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_ReadBufferFromFileDescriptorRead", + by_labels=[], + ), + legend_format="File Descriptor", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="Write flow", + description="The flow of different kinds of write operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes", + by_labels=[], + ), + legend_format="File Descriptor", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMWriteBytes", by_labels=[] + ), + legend_format="Page", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMBackgroundWriteBytes", + by_labels=[], + ), + legend_format="PageBackGround", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + fill=1, + null_point_mode="null", + decimals=1, + ), + graph_panel( + title="Read flow", + description="The flow of different kinds of read operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes", + by_labels=[], + ), + legend_format="File Descriptor", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMReadBytes", by_labels=[] + ), + legend_format="Page", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_PSMBackgroundReadBytes", + by_labels=[], + ), + legend_format="PageBackGround", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Compression Ratio", + description="The compression ratio of different compression algorithm", + targets=[ + target( + expr='sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"lz4_uncompressed_bytes"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"lz4_compressed_bytes"}[$__rate_interval]))', + legend_format="lz4", + ), + target( + expr='sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"lightweight_uncompressed_bytes"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"lightweight_compressed_bytes"}[$__rate_interval]))', + legend_format="lightweight", + ), + ], + yaxes=yaxes(left_format="short"), + legend=graph_legend(avg=True, max=False), + fill=1, + null_point_mode="null", + ), + ops_panel( + "Compression Algorithm Count", + "tiflash_storage_pack_compression_algorithm_count", + by_labels=["type"], + description="The count of the compression algorithm used by each data part", + y_left="short", + fill=1, + left_min=None, + null_point_mode="null", + panel_legend=graph_legend(max=False, total=True), + ), + ] + ) + return layout.row_panel + + +def StorageReadPoolDataSharing() -> RowPanel: + layout = Layout(title="Storage Read Pool & Data Sharing") + layout.row( + [ + ops_panel( + "Read Tasks OPS", + "tiflash_storage_read_tasks_count", + by_labels=["instance"], + description="Total number of storage engine read tasks", + ), + graph_panel( + title="Read Snapshots", + targets=[ + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SegmentReadTasks" + ), + legend_format="read_tasks-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_PSMVCCSnapshotsList" + ), + legend_format="snapshot_list-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_PSMVCCNumSnapshots" + ), + legend_format="num_snapshot-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfRead" + ), + legend_format="read-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfReadRaw" + ), + legend_format="read_raw-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfDeltaMerge" + ), + legend_format="delta_merge-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfDeltaCompact" + ), + legend_format="delta_compact-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfSegmentMerge" + ), + legend_format="seg_merge-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfSegmentSplit" + ), + legend_format="seg_split-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_current_metric_DT_SnapshotOfPlaceIndex" + ), + legend_format="place_index-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime" + ), + legend_format="max_snapshot_lifetime-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="short", + right_format="s", + left_min="0", + right_min="0", + right_show=True, + ), + fill=1, + null_point_mode="null", + series_overrides=[{"alias": "/max_snapshot_lifetime/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + duration_panel( + "Read Thread Internal Duration", + "tiflash_read_thread_internal_us", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + unit="µs", + ), + graph_panel( + title="Read Thread Scheduling", + description="The information of read thread scheduling.", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_read_thread_counter", + label_selectors=[ + 'type=~"ru_exhausted|sche_active_segment_limit|sche_from_cache|sche_new_task|sche_no_pool|sche_no_ru|sche_no_segment|sche_no_slot|push_block_bytes"' + ], + by_labels=["type"], + ), + legend_format="{{type}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="binBps", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend(current=False, max=False), + series_overrides=[{"alias": "/push_block/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Data Sharing", + description="The information of data sharing cache hit ratio. Data sharing cache is purpose-built for OLAP workload that can reduce repeated data reads of concurrent table scanning.", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_read_thread_counter", + label_selectors=['type=~"add_cache_total_bytes_limit"'], + by_labels=["type"], + ), + legend_format="{{type}}", + ), + target( + expr='(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"data_sharing_hit"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"data_sharing_hit|data_sharing_miss"}[$__rate_interval])))', + legend_format="data_sharing_cache_hit_ratio", + ), + target( + expr='(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"extra_column_hit"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"extra_column_hit|extra_column_miss"}[$__rate_interval])))', + legend_format="extra_column_cache_hit_ratio", + hide=True, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_show=True, + ), + series_overrides=[ + {"alias": "/cache_hit_ratio/", "yaxis": 2}, + {"alias": "/cache_hit_ratio/", "yaxis": 2}, + ], + ), + graph_panel( + title="Segment MergedTask", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_read_thread_gauge", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_min="0", + right_show=True, + ), + series_overrides=[{"alias": "/cache_hit_ratio/", "yaxis": 2}], + ), + duration_panel( + "Segment MergedTask Duration", + "tiflash_storage_read_thread_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + duration_panel( + "VersionChain", + "tiflash_storage_version_chain_ms", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + unit="ms", + ), + ops_panel( + "DeltaIndexError", + "tiflash_system_profile_event_DTDeltaIndexError", + by_labels=["instance"], + legend="DeltaIndexError-{{instance}}", + description="Errors of DeltaIndex", + y_left="cps", + y_right="opm", + panel_legend=graph_legend(current=False), + ), + ] + ) + return layout.row_panel + + +def PageStorage() -> RowPanel: + layout = Layout(title="PageStorage") + layout.row( + [ + graph_panel( + title="PageStorage Disk Usage", + description="The disk usage of PageStorage instances in each TiFlash node", + targets=[ + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_BlobDiskBytes" + ), + legend_format="blob_disk_size-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_BlobValidBytes", + by_labels=["instance"], + ), + legend_format="blob_valid_size-{{instance}}", + ), + target( + expr='sum((tiflash_system_asynchronous_metric_BlobValidBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}) / (tiflash_system_asynchronous_metric_BlobDiskBytes{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"})) by (instance)', + legend_format="blob_valid_rate-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_LogDiskBytes" + ), + legend_format="log_size-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="bytes", + right_format="percentunit", + left_min="0", + right_min="0", + right_max="1.1", + right_show=True, + ), + legend=graph_legend(max=False), + decimals=1, + series_overrides=[ + {"alias": "/^valid_rate/", "yaxis": 2}, + {"alias": "/size/", "linewidth": 3}, + ], + ), + graph_panel( + title="PageStorage File Num", + description="The number of files of PageStorage instances in each TiFlash node", + targets=[ + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_BlobFileNums", + by_labels=["instance"], + ), + legend_format="blob_file-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_asynchronous_metric_LogNums", + by_labels=["instance"], + ), + legend_format="log_file-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + ] + ) + layout.row( + [ + make_heatmap( + title="PageStorage WriteBatch Size", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_storage_page_write_batch_size_bucket", + label_selectors=['type="v3"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + duration_panel( + "Page write Duration", + "tiflash_storage_page_write_duration_seconds", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Page GC Tasks OPM", + targets=[ + target( + expr=expr_sum_increase( + "tiflash_storage_page_gc_count", by_labels=["type"] + ), + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="opm", left_min="0"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + fill=1, + ), + duration_panel( + "Page GC Duration", + "tiflash_storage_page_gc_duration_seconds", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Numer of Pages", + description="The number of pages of all TiFlash instance", + targets=[ + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_PagesInMem" + ), + legend_format="num_pages-{{instance}}", + ), + target( + expr=expr_simple( + "tiflash_system_asynchronous_metric_VersionedEntries" + ), + legend_format="num_entries-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + graph_panel( + title="PageStorage Pending Writers Num", + description="The num of pending writers in PageStorage", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_PSPendingWriterNum", + by_labels=["instance"], + ), + legend_format="size-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + ), + ] + ) + layout.row( + [ + graph_panel( + title="PageStorage stored bytes by type", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_page_data_by_types", by_labels=["type"] + ), + legend_format="{{type}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + fill=1, + null_point_mode="null", + decimals=1, + ), + graph_panel( + title="Number of Tables", + description="The number of tables running under different mode in DeltaTree", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_StoragePoolV2Only", + by_labels=["instance"], + ), + legend_format="V2-{{instance}}", + hide=True, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_StoragePoolV3Only", + by_labels=["instance"], + ), + legend_format="V3-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_StoragePoolMixMode", + by_labels=["instance"], + ), + legend_format="Mix-{{instance}}", + hide=True, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_StoragePoolUniPS", + by_labels=["instance"], + ), + legend_format="UniPS-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short"), + legend=graph_legend(max=False), + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="PS Command OPS By Instance", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_page_command_count", + by_labels=["instance", "type"], + ), + legend_format="{{type}}-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="opm", + left_min="0", + right_min="0", + right_show=True, + ), + series_overrides=[{"yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="PS Apply edits OPS By Instance", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_page_apply_edit_type", + by_labels=["instance", "type"], + ), + legend_format="{{type}}-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="opm", + left_min="0", + right_min="0", + right_show=True, + ), + series_overrides=[{"yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def RateLimiter() -> RowPanel: + layout = Layout(title="Rate Limiter") + layout.row( + [ + ops_panel( + "I/O Limiter Throughput", + "tiflash_storage_io_limiter", + by_labels=["type", "instance"], + description="The storage I/O limiter metrics.", + y_left="binBps", + fill=1, + left_decimals=0, + ), + graph_panel( + title="I/O Limiter Threshold", + description="Current limit bytes per second of Storage I/O limiter", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_io_limiter_curr", + by_labels=["type", "instance"], + ), + legend_format="{{type}}-{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes"), + legend=graph_legend(max=False), + fill=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="I/O Limiter Current Pending Gauge", + description="I/O Limiter current pending gauge.", + targets=[ + target( + expr=expr_avg( + "tiflash_system_current_metric_RateLimiterPendingWriteRequest", + by_labels=["instance"], + ), + legend_format="other-current-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_IOLimiterPendingBgWriteReq", + by_labels=["instance"], + ), + legend_format="bgwrite-current-{{instance}}", + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_IOLimiterPendingFgWriteReq", + by_labels=["instance"], + ), + legend_format="fgwrite-current-{{instance}}", + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_IOLimiterPendingBgReadReq", + by_labels=["instance"], + ), + legend_format="bgread-current-{{instance}}", + ), + target( + expr=expr_avg( + "tiflash_system_current_metric_IOLimiterPendingFgReadReq", + by_labels=["instance"], + ), + legend_format="fgread-current-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short", right_format="s", right_show=True), + fill=1, + null_point_mode="null", + series_overrides=[{"alias": "/pending/", "yaxis": 2}], + ), + graph_panel( + title="I/O Limiter Pending OPS", + description="The storage I/O limiter metrics.", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_io_limiter_pending_count", + by_labels=["type", "instance"], + ), + legend_format="{{type}}-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="s", + left_min="0", + left_decimals=0, + right_show=True, + ), + fill=1, + series_overrides=[{"alias": "", "yaxis": 2}], + ), + duration_panel( + "I/O Limiter Pending Duration", + "tiflash_storage_io_limiter_pending_seconds", + description="I/O Limiter pending duration.", + by_labels=["type"], + legend="{{type}}-pending-%s", + ), + ] + ) + return layout.row_panel + + +def StorageWriteStall() -> RowPanel: + layout = Layout(title="Storage Write Stall") + layout.row( + [ + duration_panel( + "Write Stall Duration", + "tiflash_storage_write_stall_duration_seconds", + description="The stall duration of write and delete range", + by_labels=["type", "instance"], + legend="%s-{{type}}-{{instance}}", + series_overrides=[{"alias": "99-delta_merge", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Write & Delta Management Throughput", + description="The throughput of write and delta's background management", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"write|ingest"'], + by_labels=[], + ), + legend_format="write+ingest", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type!~"write|ingest"'], + by_labels=[], + ), + legend_format="ManageDelta", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + ), + graph_panel( + title="Write & Delta Management Total", + description="The throughput of write and delta's background management", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"write|ingest"'], + by_labels=[], + ), + legend_format="write+ingest", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_storage_throughput_bytes", + label_selectors=['type!~"write|ingest"'], + by_labels=[], + ), + legend_format="ManageDelta", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Write Throughput By Instance", + description="The throughput of write by instance", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"write"'], + by_labels=["instance"], + ), + legend_format="write-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_storage_throughput_bytes", + label_selectors=['type=~"ingest"'], + by_labels=["instance"], + ), + legend_format="ingest-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="binBps", + right_format="bytes", + left_min="0", + right_show=True, + ), + legend=graph_legend(side_width=250), + null_point_mode="null", + decimals=1, + series_overrides=[{"alias": "/total/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Write Command OPS By Instance", + description="The total count of different kinds of commands received", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMWriteBlock", + by_labels=["instance", "type"], + ), + legend_format="write block-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum_increase( + "tiflash_storage_command_count", + by_labels=["instance", "type"], + ), + legend_format="{{type}}-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="opm", + left_min="0", + right_min="0", + right_show=True, + ), + series_overrides=[{"alias": "/delete_range|ingest/", "yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def Raft() -> RowPanel: + layout = Layout(title="Raft") + layout.row( + [ + ops_panel( + "Stale Read OPS", + "tiflash_stale_read_count", + by_labels=["instance"], + ), + ops_panel( + "Raft Read Index OPS", + "tiflash_raft_read_index_count", + by_labels=["instance"], + ), + ] + ) + layout.row( + [ + ops_panel( + "Learner Read Failures", + "tiflash_raft_learner_read_failures_count", + by_labels=["type"], + ), + ops_panel( + "Read Index Events", + "tiflash_raft_read_index_events_count", + by_labels=["type"], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Raft Wait Index Duration", + targets=[ + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)', + legend_format="max {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.9999", + "tiflash_raft_wait_index_duration_seconds", + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="9999 {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.999", + "tiflash_raft_wait_index_duration_seconds", + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="999 {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_raft_wait_index_duration_seconds", + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="99 {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.80", + "tiflash_raft_wait_index_duration_seconds", + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="80 {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_avg( + "tiflash_raft_wait_index_duration_seconds", + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="avg {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_sum_increase( + "tiflash_system_profile_event_RaftWaitIndexTimeout", + by_labels=["instance"], + ), + legend_format="{{instance}}-timeout", + ), + ], + yaxes=yaxes( + left_format="s", right_format="opm", left_min="0", right_show=True + ), + fill=1, + tooltip_sort=2, + series_overrides=[{"alias": "/timeout/", "yaxis": 2}], + ), + duration_panel( + "Raft Batch Read Index Duration", + "tiflash_raft_read_index_duration_seconds", + description="The number of currently applying snapshots.", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Apply Raft write logs Duration", + description="Duration of applying Raft write logs", + targets=[ + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)', + legend_format="max-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.9999", + "tiflash_raft_apply_write_command_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="9999-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.999", + "tiflash_raft_apply_write_command_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="999-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_raft_apply_write_command_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="99-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.80", + "tiflash_raft_apply_write_command_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="80-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_avg( + "tiflash_raft_apply_write_command_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="avg-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr='(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="write"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="write"}[$__rate_interval])))', + legend_format="avg-write", + ), + target( + expr='(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="admin"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="admin"}[$__rate_interval])))', + legend_format="avg-admin", + ), + target( + expr='(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="flush_region"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="flush_region"}[$__rate_interval])))', + legend_format="avg-flush_region", + ), + target( + expr='(sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="decode"}[$__rate_interval])) / sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="decode"}[$__rate_interval])))', + legend_format="avg-decode", + ), + ], + yaxes=yaxes(left_format="s", left_min="0"), + fill=1, + tooltip_sort=2, + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Region write Duration (decode)", + description='Duration of decoding Region data into blocks when writing Region data to the storage layer. (Mixed with "write logs" and "apply Snapshot" operations)', + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_data_to_storage_duration_seconds_bucket", + label_selectors=['type="decode"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Region write Duration (write blocks)", + description='Duration of writing Region data blocks to the storage layer (Mixed with "write logs" and "apply Snapshot" operations)', + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_data_to_storage_duration_seconds_bucket", + label_selectors=['type="write"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Apply Raft write logs Duration [Heatmap]", + description="Duration of applying Raft write logs", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_apply_write_command_duration_seconds_bucket", + label_selectors=['type="write"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Apply Raft admin logs Duration [Heatmap]", + description="Duration of applying Raft write logs", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_apply_write_command_duration_seconds_bucket", + label_selectors=['type="admin"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + ops_panel( + "Raft Events QPS", + "tiflash_raft_raft_events_count", + by_labels=["type"], + ), + ops_panel( + "Raft Frequent Events QPS", + "tiflash_raft_raft_frequent_events_count", + by_labels=["type"], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Raft Log Gap Heatmap", + y_format="none", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_raft_log_gap_count_bucket", + label_selectors=['type=~"applied_index"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + target( + expr=expr_sum_delta( + "tiflash_raft_raft_log_gap_count_bucket", + label_selectors=['type=~"compact_index"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Raft Entry Batch Size Heatmap", + y_format="none", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_entry_size_bucket", + label_selectors=['type=~"normal"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Region Size (by event) Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_region_flush_bytes_bucket", + label_selectors=['type=~"unflushed"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + target( + expr=expr_sum_delta( + "tiflash_raft_region_flush_bytes_bucket", + label_selectors=['type=~"flushed"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + hide=True, + ), + ], + ), + make_heatmap( + title="Big Write To Region Size Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_flow_bytes_bucket", + label_selectors=['type=~"big_write_to_region"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Write Committed Size Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_flow_bytes_bucket", + label_selectors=['type=~"write_committed"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + ops_panel( + "Raft Eager GC OPS", + "tiflash_raft_eager_gc_count", + by_labels=["type"], + ), + duration_panel( + "Raft Eager GC Duration", + "tiflash_raft_eager_gc_duration_seconds", + description="Duration of Raft logs eager GC tasks", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + ops_panel( + "Keys flow", + "tiflash_raft_process_keys", + by_labels=["type"], + description="The keys flow of different kinds of Raft operations", + y_left="short", + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + ops_panel( + "Raft throughput", + "tiflash_raft_throughput_bytes", + by_labels=["type"], + y_left="short", + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Upstream Latency [Heatmap]", + description="Latency that TiKV sends raft log to TiFlash.", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_upstream_latency_bucket", by_labels=["le"] + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + duration_panel( + "Upstream Latency", + "tiflash_raft_upstream_latency", + description="Latency that TiKV sends raft log to TiFlash.", + show_avg=True, + ), + ] + ) + layout.half_row( + [ + graph_panel( + title="Log Replication Rejected", + targets=[ + target( + expr='sum(rate(tiflash_proxy_tikv_server_raft_append_rejects{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$tiflash_role"}[$__rate_interval])) by (instance)', + legend_format="{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + ), + ] + ) + return layout.row_panel + + +def RaftSnapshotIngestSST() -> RowPanel: + layout = Layout(title="Raft Snapshot / IngestSST") + layout.row( + [ + duration_panel( + "Heavy Raft Apply Duration", + "tiflash_raft_command_duration_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Applying snapshots Count", + description="The number of currently applying snapshots.", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_RaftNumSnapshotsPendingApply", + by_labels=["instance"], + ), + legend_format="Pending-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_RaftNumPrehandlingSubTasks", + by_labels=["instance"], + ), + legend_format="PrehandleSubtasks-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_RaftNumParallelPrehandlingTasks", + by_labels=["instance"], + ), + legend_format="ParallelTasks-{{instance}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_RaftNumWaitedParallelPrehandlingTasks", + by_labels=["instance"], + ), + legend_format="Pending-ParallelTasks-{{instance}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + fill=1, + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Snapshot Uncommitted Size Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_flow_bytes_bucket", + label_selectors=['type=~"snapshot_uncommitted"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ops_panel( + "Ongoing raft snapshot", + "tiflash_raft_ongoing_snapshot_total_bytes", + by_labels=["type"], + legend="{{le}}", + y_left="bytes", + fill=1, + left_min=None, + null_point_mode="null", + panel_legend=graph_legend( + current=False, max=False, align_as_table=False, right_side=False + ), + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Snapshot Size Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_snapshot_total_bytes_bucket", + label_selectors=['type="approx_raft_snapshot"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Snapshot Predecode Duration", + description="Duration of pre-decode when applying region snapshot", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_command_duration_seconds_bucket", + label_selectors=['type="snapshot_predecode"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Snapshot Prehandle Throughput Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_command_throughput_seconds_bucket", + label_selectors=['type="prehandle_snapshot"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Snapshot Flush Duration", + description="Duration of pre-decode when applying region snapshot", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_command_duration_seconds_bucket", + label_selectors=['type="snapshot_flush"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.row( + [ + make_heatmap( + title="Ingest Uncommitted Size Heatmap", + y_format="bytes", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_write_flow_bytes_bucket", + label_selectors=['type=~"ingest_uncommitted"'], + by_labels=["le", "type"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + make_heatmap( + title="Snapshot Predecode SST to DT Duration", + description="Duration of SST to DT in pre-decode when applying region snapshot", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_command_duration_seconds_bucket", + label_selectors=['type="snapshot_predecode_sst2dt"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + layout.half_row( + [ + make_heatmap( + title="Ingest SST Duration", + description="Duration of ingesting SST", + y_format="s", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_raft_command_duration_seconds_bucket", + label_selectors=['type="ingest_sst"'], + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + return layout.row_panel + + +def RoughSetFilterRateHistogram() -> RowPanel: + layout = Layout(title="Rough Set Filter Rate Histogram") + layout.row( + [ + graph_panel( + title="Rough Set Filter Rate", + targets=[ + target( + expr='avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (instance)', + legend_format="1min-{{instance}}", + interval_factor=1, + ), + target( + expr='avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (instance)', + legend_format="5min-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMFileFilterNoFilter", + by_labels=["instance"], + ), + legend_format="No Filter-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMFileFilterAftPKAndPackSet", + by_labels=["instance"], + ), + legend_format="PK Filter-{{instance}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_DMFileFilterAftRoughSet", + by_labels=["instance"], + ), + legend_format="RS Filter-{{instance}}", + hide=True, + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="percentunit", + right_format="short", + left_min="0", + right_show=True, + ), + legend=graph_legend(max=False), + series_overrides=[ + {"alias": "/^RS Filter/", "yaxis": 2}, + {"alias": "/^PK/", "yaxis": 2}, + {"alias": "/^No Filter/", "yaxis": 2}, + ], + ), + make_heatmap( + title="Rough Set Filter Rate Histogram", + y_format="percent", + log_base=1, + hide_zero_buckets=True, + max_data_points=512, + targets=[ + target( + expr=expr_sum_delta( + "tiflash_storage_rough_set_filter_rate_bucket", + by_labels=["le"], + ), + legend_format="{{le}}", + interval_factor=2, + ), + ], + ), + ] + ) + return layout.row_panel + + +def DisaggregatedWrite() -> RowPanel: + layout = Layout(title="Disaggregated-Write") + layout.row( + [ + duration_panel( + "Checkpoint Upload Duration", + "tiflash_storage_checkpoint_seconds", + description="PageStorage Checkpoint Duration", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + ), + graph_panel( + title="Checkpoint Upload flow", + description="The flow of checkpoint operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_checkpoint_flow", + label_selectors=['type="incremental"'], + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="incremental {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_storage_checkpoint_flow", + label_selectors=['type="compaction"'], + by_labels=[ADDITIONAL_GROUPBY], + ), + legend_format="compaction {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + ops_panel( + "Checkpoint Upload keys speed by type (all)", + "tiflash_storage_checkpoint_keys_by_types", + by_labels=["type", ADDITIONAL_GROUPBY], + legend="{{type}} {{$additional_groupby}}", + description="The keys of checkpoint operations. All keys are uploaded in the checkpoint. Grouped by key types.", + fill=1, + ), + ops_panel( + "Checkpoint Upload flow by type (incremental+compaction)", + "tiflash_storage_checkpoint_flow_by_types", + by_labels=["type", ADDITIONAL_GROUPBY], + legend="{{type}} {{$additional_groupby}}", + description="The flow of checkpoint operations. Group by key types", + y_left="binBps", + fill=1, + null_point_mode="null", + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote File Num", + description="The number of files of owned by each TiFlash node", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_remote_stats", + label_selectors=['type="num_files"'], + by_labels=["instance"], + ), + legend_format="checkpoint_data-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + graph_panel( + title="Remote Store Usage", + description="The remote store usage owned by each TiFlash node", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_remote_stats", + label_selectors=['type="total_size"'], + by_labels=["instance"], + ), + legend_format="remote_size-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_storage_remote_stats", + label_selectors=['type="valid_size"'], + by_labels=["instance"], + ), + legend_format="valid_size-{{instance}}", + ), + target( + expr='sum((tiflash_storage_remote_stats{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="valid_size"}) / (tiflash_storage_remote_stats{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="total_size"})) by (instance)', + legend_format="valid_rate-{{instance}}", + hide=True, + ), + ], + yaxes=yaxes( + left_format="bytes", + right_format="percentunit", + left_min="0", + right_min="0", + right_max="1.1", + right_show=True, + ), + legend=graph_legend(max=False), + decimals=1, + series_overrides=[ + {"alias": "/^valid_rate/", "yaxis": 2}, + {"alias": "/size/", "linewidth": 3}, + ], + ), + ] + ) + layout.row( + [ + ops_panel( + "Remote Object Lock Request QPS", + "tiflash_disaggregated_object_lock_request_count", + by_labels=["type", ADDITIONAL_GROUPBY], + legend="{{type}} {{$additional_groupby}}", + y_left="none", + ), + duration_panel( + "Remote Object Lock Duration", + "tiflash_disaggregated_object_lock_request_duration_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote Store Summary", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_s3_store_summary_bytes", + by_labels=["instance", "store_id", "type"], + ), + legend_format="store-{{store_id}}-{{type}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + duration_panel( + "Remote GC Duration Breakdown", + "tiflash_storage_s3_gc_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + series_overrides=[ + {"alias": "/total/", "yaxis": 2}, + {"alias": "/one_store/", "yaxis": 2}, + {"alias": "/clean_locks/", "yaxis": 2}, + ], + ), + graph_panel( + title="Remote GC Status", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_s3_gc_status", + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Local Lock Manager status", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_s3_lock_mgr_status", + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + legend=graph_legend(max=False), + decimals=1, + ), + ops_panel( + "Local Lock Manager QPS", + "tiflash_storage_s3_lock_mgr_counter", + by_labels=["type", ADDITIONAL_GROUPBY], + legend="{{type}} {{$additional_groupby}}", + y_left="none", + ), + ] + ) + layout.row( + [ + graph_panel( + title="FAP result", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_fap_task_result", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend(max=False), + series_overrides=[{"alias": "/hit_ratio/", "yaxis": 2}], + ), + graph_panel( + title="FAP state", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_fap_task_state", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend(max=False), + series_overrides=[{"alias": "/hit_ratio/", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + duration_panel( + "FAP time by stage", + "tiflash_fap_task_duration_seconds", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + series_overrides=[{"alias": "/hit_ratio/", "yaxis": 2}], + ), + graph_panel( + title="FAP no match reason", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_fap_nomatch_reason", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend(max=False), + series_overrides=[{"alias": "/hit_ratio/", "yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def DisaggregatedCompute() -> RowPanel: + layout = Layout(title="Disaggregated-Compute") + layout.row( + [ + duration_panel( + "Read Duration Breakdown", + "tiflash_disaggregated_breakdown_duration_seconds", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote Cache Operations", + description="Remote Cache Operations", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_remote_cache", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + ), + target( + expr='(sum(rate(tiflash_storage_remote_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dtfile_hit"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"dtfile_hit|dtfile_miss"}[$__rate_interval])))', + legend_format="dtfile_cache_hit_ratio", + ), + target( + expr='(sum(rate(tiflash_storage_remote_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"page_hit"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"page_hit|page_miss"}[$__rate_interval])))', + legend_format="page_cache_hit_ratio", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_show=True, + ), + series_overrides=[ + {"alias": "dtfile_cache_hit_ratio", "yaxis": 2}, + {"alias": "page_cache_hit_ratio", "yaxis": 2}, + ], + ), + ops_panel( + "Remote Cache Flow", + "tiflash_storage_remote_cache_bytes", + by_labels=["type", ADDITIONAL_GROUPBY], + legend="{{type}} {{$additional_groupby}}", + description="Remote Cache Flow", + y_left="binBps", + ), + ] + ) + layout.row( + [ + duration_panel( + "Remote Cache BG Download Duration", + "tiflash_storage_remote_cache_bg_download_stage_seconds", + by_labels=["stage", "file_type"], + legend="%s-{{stage}}-{{file_type}} {{$additional_groupby}}", + ), + duration_panel( + "Remote Cache Wait on Downloading Duration", + "tiflash_storage_remote_cache_wait_on_downloading_seconds", + by_labels=["result", "file_type"], + legend="%s-{{result}}-{{file_type}} {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote Cache Wait on Downloading OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_remote_cache_wait_on_downloading_result", + by_labels=["result", "file_type", ADDITIONAL_GROUPBY], + ), + legend_format="{{result}}-{{file_type}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="s", + left_min="0", + left_decimals=0, + right_show=True, + ), + fill=1, + series_overrides=[{"alias": "", "yaxis": 2}], + ), + ops_panel( + "Remote Cache Wait on Downloading Flow", + "tiflash_storage_remote_cache_wait_on_downloading_bytes", + by_labels=["result", "file_type", ADDITIONAL_GROUPBY], + legend="{{result}}-{{file_type}} {{$additional_groupby}}", + y_left="binBps", + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote Cache Gauge", + targets=[ + target( + expr=expr_sum( + "tiflash_storage_remote_cache_status", + by_labels=["type", "instance"], + ), + legend_format="{{type}}-{{instance}}", + ), + ], + yaxes=yaxes(left_format="short"), + fill=1, + ), + graph_panel( + title="Remote Cache Reject Download Type OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_remote_cache_reject", + by_labels=["reason", "file_type", ADDITIONAL_GROUPBY], + ), + legend_format="{{reason}}-{{file_type}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="s", + left_min="0", + left_decimals=0, + right_show=True, + ), + fill=1, + series_overrides=[{"alias": "", "yaxis": 2}], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Remote Cache Usage", + description="Remote Cache Usage", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_DTFileCacheCapacity", + by_labels=["instance"], + ), + legend_format="DTFileCapacity-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_DTFileCacheUsed", + by_labels=["instance"], + ), + legend_format="DTFileUsed-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_PageCacheCapacity", + by_labels=["instance"], + ), + legend_format="PageCapacity-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_PageCacheUsed", + by_labels=["instance"], + ), + legend_format="PageUsed-{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(max=False), + ), + graph_panel( + title="Memory Usage of Storage Tasks", + description="Memory Usage of Storage Tasks", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_MemoryTrackingQueryStorageTask", + by_labels=["instance"], + ), + legend_format="MemoryTrackingQueryStorageTask-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_MemoryTrackingFetchPages", + by_labels=["instance"], + ), + legend_format="MemoryTrackingFetchPages-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_DT_DeltaIndexCacheSize", + by_labels=["instance"], + ), + legend_format="DeltaIndexCacheSize-{{instance}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_MemoryTrackingSharedColumnData", + by_labels=["instance"], + ), + legend_format="SharedColumnData-{{instance}}", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0"), + legend=graph_legend(current=False), + ), + ] + ) + layout.row( + [ + graph_panel( + title="MVCCIndexCache", + description="DeltaIndex cache of ReadNodes", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_mvcc_index_cache", + by_labels=["type", "instance"], + ), + legend_format="{{type}}-{{instance}}", + ), + target( + expr='(sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type=~"hit"}[$__rate_interval])) by (instance) / sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) by (instance))', + legend_format="hit_ratio-{{instance}}", + ), + ], + yaxes=yaxes( + left_format="ops", + right_format="percentunit", + left_min="0", + right_show=True, + ), + series_overrides=[{"alias": "/hit_ratio/", "yaxis": 2}], + ), + duration_panel( + "PlaceIndex Tasks Duration", + "tiflash_storage_subtask_duration_seconds", + description="Duration of storage's internal sub tasks", + by_labels=["type"], + legend="%s-{{type}} {{$additional_groupby}}", + extra_label_selectors=['type="place_index_update"'], + ), + ] + ) + layout.row( + [ + graph_panel( + title="PlaceIndexTask/Reuse OPS", + description="Total number of storage's internal sub tasks", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_storage_place_index_count", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_storage_subtask_count", + label_selectors=['type=~"place_index_update"'], + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="{{type}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0", left_decimals=1), + ), + graph_panel( + title="PlaceIndex update rows/deletes", + targets=[ + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_place_index_stats_count_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)', + legend_format="max {{$additional_groupby}}", + hide=True, + interval_factor=1, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_storage_place_index_stats_count", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="99-{{type}} {{$additional_groupby}}", + hide=True, + interval_factor=1, + ), + target( + expr='sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval])) / sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))', + legend_format="avg-{{type}} {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short", left_min="0"), + fill=1, + ), + ] + ) + return layout.row_panel + + +def S3() -> RowPanel: + layout = Layout(title="S3") + layout.row( + [ + graph_panel( + title="S3 Bytes", + description="S3 read/write throughput", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteBytes", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3WriteBytes {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadBytes", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3ReadBytes {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteDMFileBytes", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3WriteDMFileBytes {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="binBps", left_min="0"), + ), + graph_panel( + title="S3 OPS", + description="S3 OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3PutObject", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3PutObject {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3GetObject", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3GetObject {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3HeadObject", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3HeadObject {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ListObjects", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3ListObjects {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3DeleteObject", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3DeleteObject {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3CopyObject", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3CopyObject {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3CreateMultipartUpload", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3CreateMultipartUpload {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3UploadPart", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3UploadPart {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3CompleteMultipartUpload", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3CompleteMultipartUpload {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3PutDMFile", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3PutDMFile {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IORead", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IORead {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeek", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeek {{$additional_groupby}}", + hide=True, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + ] + ) + layout.row( + [ + graph_panel( + title="S3 Retry OPS", + description="S3 Retry OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3GetObjectRetry", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3GetObjectRetry {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3PutObjectRetry", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3PutObjectRetry {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3PutDMFileRetry", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3PutDMFileRetry {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOReadError", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOReadError {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeekError", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeekError {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeekBackward", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeekBackward {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + duration_panel( + "S3 Request Duration", + "tiflash_storage_s3_request_seconds", + description="S3 Request Duration", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="S3 HTTP OPS", + description="S3 HTTP OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadRequestsCount", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="read-count {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteRequestsCount", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="write-count {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadRequestsErrors", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="read-error {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteRequestsErrors", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="write-error {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadRequestsThrottling", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="read-throttling {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteRequestsThrottling", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="write-throttling {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadRequestsRedirects", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="read-redirects {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteRequestsRedirects", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="write-redirects {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3ReadRequestsNotFound", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="read-notfound {{$additional_groupby}}", + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3WriteRequestsNotFound", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="write-notfound {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + duration_panel( + "S3 HTTP Request Duration", + "tiflash_storage_s3_http_request_seconds", + description="S3 HTTP Request Duration", + by_labels=["type"], + legend="{{type}}-%s {{$additional_groupby}}", + ), + ] + ) + layout.row( + [ + graph_panel( + title="S3 on-going instances", + description="S3 HTTP OPS", + targets=[ + target( + expr=expr_sum( + "tiflash_system_current_metric_S3Requests", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3Requests {{$additional_groupby}}", + ), + target( + expr=expr_sum( + "tiflash_system_current_metric_S3RandomAccessFile", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3RandomAccessFile {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + graph_panel( + title="S3RandomAccessFile OPS", + targets=[ + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOReadError", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOReadError {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeekError", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeekError {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeekBackward", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeekBackward {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IORead", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IORead {{$additional_groupby}}", + interval_factor=1, + ), + target( + expr=expr_sum_rate( + "tiflash_system_profile_event_S3IOSeek", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="S3IOSeek {{$additional_groupby}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + ), + ] + ) + return layout.row_panel + + +def PipelineModel() -> RowPanel: + layout = Layout(title="Pipeline Model") + layout.row( + [ + graph_panel( + title="Task Thread Pool Size", + targets=[ + target( + expr=expr_max( + "tiflash_pipeline_scheduler", + label_selectors=['type=~".*_task_thread_pool_size"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + graph_panel( + title="Task Count", + targets=[ + target( + expr=expr_max( + "tiflash_pipeline_scheduler", + label_selectors=['type=~".*_tasks_count"'], + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_pipeline_scheduler", + label_selectors=['type=~".*_tasks_count"'], + by_labels=["type"], + ), + legend_format="sum({{type}})", + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + layout.row( + [ + ops_panel( + "Task Status Change OPS", + "tiflash_pipeline_task_change_to_status", + by_labels=["type"], + y_left="none", + ), + graph_panel( + title="Task Duration", + targets=[ + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)', + legend_format="max-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.9999", + "tiflash_pipeline_task_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="9999-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.999", + "tiflash_pipeline_task_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="999-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_pipeline_task_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="99-{{type}} {{$additional_groupby}}", + ), + target( + expr=expr_histogram_quantile( + "0.80", + "tiflash_pipeline_task_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="80-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr=expr_histogram_avg( + "tiflash_pipeline_task_duration_seconds", + by_labels=["type", ADDITIONAL_GROUPBY], + ), + legend_format="avg-{{type}} {{$additional_groupby}}", + hide=True, + ), + target( + expr='(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu_execute"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu_execute"}[$__rate_interval])))', + legend_format="avg-cpu_execute", + ), + target( + expr='(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu_queue"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu_queue"}[$__rate_interval])))', + legend_format="avg-cpu_queue", + ), + target( + expr='(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io_execute"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io_execute"}[$__rate_interval])))', + legend_format="avg-io_execute", + ), + target( + expr='(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io_queue"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io_queue"}[$__rate_interval])))', + legend_format="avg-io_queue", + ), + target( + expr='(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="await"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="await"}[$__rate_interval])))', + legend_format="avg-await", + ), + ], + yaxes=yaxes(left_format="s", left_min="0"), + fill=1, + tooltip_sort=2, + ), + ] + ) + layout.row( + [ + graph_panel( + title="Task Max Execute Time Per Round", + targets=[ + target( + expr=expr_histogram_quantile( + "0.95", + "tiflash_pipeline_task_execute_max_time_seconds_per_round", + by_labels=["type"], + ), + legend_format="95-{{type}}", + ), + target( + expr=expr_histogram_quantile( + "0.95", + "tiflash_pipeline_task_execute_max_time_seconds_per_round", + by_labels=["type"], + ), + legend_format="99-{{type}}", + ), + target( + expr=expr_histogram_quantile( + "0.99", + "tiflash_pipeline_task_execute_max_time_seconds_per_round", + by_labels=["type"], + ), + legend_format="999-{{type}}", + ), + target( + expr='histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role"}[$__rate_interval]))) by (le, type) / 1000000000)', + legend_format="100-{{type}}", + ), + target( + expr='sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="cpu"}[$__rate_interval]))', + legend_format="avg-cpu", + ), + target( + expr='sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="io"}[$__rate_interval]))', + legend_format="avg-io", + ), + ], + yaxes=yaxes(left_format="s", left_min="0"), + fill=1, + ), + graph_panel( + title="Threads CPU of CPU Task Thread Pool", + targets=[ + target( + expr='sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"cpu_pool", instance=~"$tiflash_role"}[$__rate_interval])) by (instance)', + legend_format="{{name}} {{instance}}", + ), + target( + expr='count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"cpu_pool", instance=~"$tiflash_role"}) by (instance)', + legend_format="Limit", + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0"), + null_point_mode="null", + series_overrides=[ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + "nullPointMode": "connected", + } + ], + ), + ] + ) + layout.row( + [ + graph_panel( + title="Threads CPU of IO Task Thread Pool", + targets=[ + target( + expr='sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"io_pool", instance=~"$tiflash_role"}[$__rate_interval])) by (instance)', + legend_format="{{name}} {{instance}}", + ), + target( + expr='count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"io_pool", instance=~"$tiflash_role"}) by (instance)', + legend_format="Limit", + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0"), + null_point_mode="null", + series_overrides=[ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + "nullPointMode": "connected", + } + ], + ), + graph_panel( + title="Threads CPU of Wait Reactor", + targets=[ + target( + expr='sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"WaitReactor", instance=~"$tiflash_role"}[$__rate_interval])) by (instance)', + legend_format="{{name}} {{instance}}", + ), + target( + expr='count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", name=~"WaitReactor", instance=~"$tiflash_role"}) by (instance)', + legend_format="Limit", + ), + ], + yaxes=yaxes(left_format="percentunit", left_min="0"), + null_point_mode="null", + series_overrides=[ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": True, + "legend": False, + "linewidth": 2, + "nullPointMode": "connected", + } + ], + ), + ] + ) + layout.half_row( + [ + graph_panel( + title="Wait notify task details", + description="wait notify task details", + targets=[ + target( + expr=expr_max( + "tiflash_pipeline_wait_on_notify_tasks", + by_labels=["instance", "type"], + ), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + target( + expr=expr_sum( + "tiflash_pipeline_wait_on_notify_tasks", by_labels=["type"] + ), + legend_format="sum({{type}})", + ), + ], + yaxes=yaxes(left_format="none", left_min="0"), + ), + ] + ) + return layout.row_panel + + +def TiFlashResourceControl() -> RowPanel: + layout = Layout(title="TiFlash Resource Control") + layout.row( + [ + graph_panel( + title="TiFlash Resource Group", + description="Metas of resource group", + targets=[ + target( + expr=expr_max( + "tiflash_resource_group", + label_selectors=['type="remaining_tokens"'], + by_labels=["instance", "resource_group"], + ), + legend_format="remaining_tokens-{{instance}}-{{resource_group}}", + ), + target( + expr=expr_max( + "tiflash_resource_group", + label_selectors=['type="avg_speed"'], + by_labels=["instance", "resource_group"], + ), + legend_format="avg_speed-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_resource_group_counter", + label_selectors=['type="total_consumption"'], + by_labels=["instance", "resource_group"], + ), + legend_format="total_consumption-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_max( + "tiflash_resource_group", + label_selectors=['type="bucket_fill_rate"'], + by_labels=["instance", "resource_group"], + ), + legend_format="bucket_fill_rate-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_max( + "tiflash_resource_group", + label_selectors=['type="bucket_capacity"'], + by_labels=["instance", "resource_group"], + ), + legend_format="bucket_capacity-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_resource_group_counter", + label_selectors=['type="request_gac_count"'], + by_labels=["instance", "resource_group"], + ), + legend_format="request_gac_count-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_resource_group_counter", + label_selectors=['type="gac_req_ru_consumption_delta"'], + by_labels=["instance", "resource_group"], + ), + legend_format="gac_req_ru_consumption_delta-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_resource_group_counter", + label_selectors=['type="compute_ru_consumption"'], + by_labels=["instance", "resource_group"], + ), + legend_format="compute_ru_consumption-{{instance}}-{{resource_group}}", + hide=True, + ), + target( + expr=expr_sum_rate( + "tiflash_resource_group_counter", + label_selectors=['type="storage_ru_consumption"'], + by_labels=["instance", "resource_group"], + ), + legend_format="storage_ru_consumption-{{instance}}-{{resource_group}}", + hide=True, + ), + ], + yaxes=yaxes(left_format="short"), + legend=graph_legend(avg=True, max=False), + fill=1, + null_point_mode="null", + ), + graph_panel( + title="Request Unit", + description="Request Unit for tidb-serverless charging", + targets=[ + target( + expr='sum(rate(tiflash_storage_sync_replica_ru{instance=~"$tiflash_role"}[$__rate_interval])) by (keyspace_id, $additional_groupby)', + legend_format="replica-sync-rate-{{keyspace_id}}", + ), + target( + expr='sum(increase(tiflash_storage_sync_replica_ru{instance=~"$tiflash_role"}[24h])) by (keyspace_id, $additional_groupby)', + legend_format="replica-sync-sum-24h-{{keyspace_id}} {{$additional_groupby}}", + ), + target( + expr='sum(rate(tiflash_compute_request_unit{instance=~"$tiflash_role"}[$__rate_interval])) by (cluster_id, $additional_groupby)', + legend_format="query-rate-{{cluster_id}} {{$additional_groupby}}", + ), + target( + expr='sum(increase(tiflash_compute_request_unit{instance=~"$tiflash_role"}[24h])) by (cluster_id, $additional_groupby)', + legend_format="query-sum-24h-{{cluster_id}} {{$additional_groupby}}", + ), + target( + expr='sum(rate(tiflash_storage_ru_read_bytes{instance=~"$tiflash_role"}[$__rate_interval])) by (keyspace, resource_group, type, $additional_groupby) / (64 * 1024)', + legend_format="storage-{{keyspace}}_{{resource_group}}_{{type}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes( + left_format="cps", + right_format="short", + left_min="0", + right_min="0", + right_show=True, + ), + legend=graph_legend(max=False), + decimals=1, + series_overrides=[{"alias": "/sum/", "yaxis": 2}], + ), + ] + ) + return layout.row_panel + + +def StatusServer() -> RowPanel: + layout = Layout(title="Status Server") + layout.row( + [ + duration_panel( + "Status API Request Duration", + "tiflash_proxy_tikv_status_server_proxy_request_duration_seconds", + by_labels=["path"], + legend="%s-{{path}} {{$additional_groupby}}", + instance_selector=PROXY_LABEL_SELECTORS, + ), + graph_panel( + title="Status API Request (op/s)", + targets=[ + target( + expr='sum(rate( tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count {k8s_cluster="$k8s_cluster",tidb_cluster="$tidb_cluster", instance=~"$proxy_instance", instance=~"$tiflash_role"} [$__rate_interval] )) by (path, $additional_groupby)', + legend_format="{{path}} {{$additional_groupby}}", + ), + ], + yaxes=yaxes(left_format="ops", left_min="0"), + fill=1, + null_point_mode="null", + ), + ] + ) + return layout.row_panel + + +def VectorSearch() -> RowPanel: + layout = Layout(title="Vector Search") + layout.row( + [ + graph_panel( + title="In-Memory Vector Index Instances", + targets=[ + target( + expr='sum by (type, instance) ( tiflash_vector_index_active_instances{ k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role" } )', + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="short", left_min="0", left_decimals=0), + legend=graph_legend(current=False), + decimals=0, + ), + graph_panel( + title="Vector Index Estimated Memory Usage", + targets=[ + target( + expr=expr_simple("tiflash_vector_index_memory_usage"), + legend_format="{{instance}}-{{type}}", + interval_factor=1, + ), + target( + expr=expr_simple( + "tiflash_process_rss_by_type_bytes", + label_selectors=['type="file"'], + ), + legend_format="{{instance}}-RssFile", + ), + ], + yaxes=yaxes(left_format="bytes", left_min="0", left_decimals=0), + legend=graph_legend(current=False), + decimals=0, + ), + ] + ) + layout.row( + [ + graph_panel( + title="99.9% Vector Search Duration (Per Request)", + targets=[ + target( + expr='histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type!="build" } [$__rate_interval] )) by (le, type) )', + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes( + left_format="s", + right_format="s", + left_min="0", + left_decimals=1, + right_min="0", + right_decimals=1, + right_show=True, + ), + legend=graph_legend(current=False), + decimals=1, + series_overrides=[{"alias": "/download/", "yaxis": 2}], + ), + graph_panel( + title="99.9% Vector Index Build Duration (Per DMFile Column)", + targets=[ + target( + expr='histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", instance=~"$tiflash_role", type="build" } [$__rate_interval] )) by (le, type) )', + legend_format="{{type}}", + interval_factor=1, + ), + ], + yaxes=yaxes(left_format="s", left_min="0", left_decimals=1), + legend=graph_legend(current=False), + decimals=1, + ), + ] + ) + return layout.row_panel + + +dashboard = Dashboard( + title="Test-Cluster-TiFlash-Summary", + uid="SVbh2xUWk", + timezone="browser", + refresh="1m", + inputs=[DATASOURCE_INPUT], + editable=True, + templating=Templates(), + panels=[ + Server(), + ThreadsCPU(), + Threads(), + Coprocessor(), + TaskScheduler(), + DDL(), + ImbalanceReadWrite(), + MemoryTrace(), + ColumnarStorage(), + Storage(), + StorageReadPoolDataSharing(), + PageStorage(), + RateLimiter(), + StorageWriteStall(), + Raft(), + RaftSnapshotIngestSST(), + RoughSetFilterRateHistogram(), + DisaggregatedWrite(), + DisaggregatedCompute(), + S3(), + PipelineModel(), + TiFlashResourceControl(), + StatusServer(), + VectorSearch(), + ], + # Set 14 or larger to support shared crosshair or shared tooltip. + # See https://github.com/grafana/grafana/blob/v10.2.2/public/app/features/dashboard/state/DashboardMigrator.ts#L443-L445 + schemaVersion=14, + graphTooltip=GRAPH_TOOLTIP_MODE_SHARED_CROSSHAIR, +).auto_panel_ids() diff --git a/metrics/grafana/tiflash_summary.json b/metrics/grafana/tiflash_summary.json index 799643c2e3f..a93937d4e31 100644 --- a/metrics/grafana/tiflash_summary.json +++ b/metrics/grafana/tiflash_summary.json @@ -1,96 +1,91 @@ { "__inputs": [ { - "name": "DS_TEST-CLUSTER", - "label": "Test-Cluster", "description": "", - "type": "datasource", + "label": "Test-Cluster", + "name": "DS_TEST-CLUSTER", "pluginId": "prometheus", - "pluginName": "Prometheus" - } - ], - "__requires": [ - { - "type": "grafana", - "id": "grafana", - "name": "Grafana", - "version": "7.5.11" - }, - { - "type": "panel", - "id": "graph", - "name": "Graph", - "version": "" - }, - { - "type": "panel", - "id": "heatmap", - "name": "Heatmap", - "version": "" - }, - { - "type": "datasource", - "id": "prometheus", - "name": "Prometheus", - "version": "1.0.0" + "pluginName": "Prometheus", + "type": "datasource" } ], "annotations": { - "list": [ - { - "builtIn": 1, - "datasource": "${DS_TEST-CLUSTER}", - "enable": true, - "hide": true, - "iconColor": "rgba(0, 211, 255, 1)", - "name": "Annotations & Alerts", - "type": "dashboard" - } - ] + "list": [] }, + "description": "", "editable": true, "gnetId": null, "graphTooltip": 1, + "hideControls": false, "id": null, - "iteration": 1783577500370, "links": [], "panels": [ { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, "y": 0 }, - "id": 4, + "height": null, + "hideTimeOverride": false, + "id": 1, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, + "decimals": 3, "description": "The storage size per TiFlash instance.\n(Not including some disk usage of TiFlash-Proxy by now)", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 5, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 0, - "y": 1 + "y": 0 }, - "hiddenSeries": false, - "id": 53, + "height": null, + "hideTimeOverride": false, + "id": 2, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, @@ -110,53 +105,74 @@ "lines": true, "linewidth": 0, "links": [], +<<<<<<< HEAD "nullPointMode": "null", +======= + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", +>>>>>>> 08be6d5d0a (metrics: migrate TiFlash Summary dashboard generation to grafanalib (#11027)) "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": true, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_StoreSizeUsed{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoreSizeUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-local", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoreSizeUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(tiflash_system_current_metric_StoreSizeUsedRemote{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoreSizeUsedRemote\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-remote", - "refId": "B", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoreSizeUsedRemote\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Store size", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -164,6 +180,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -172,6 +189,7 @@ "show": true }, { +<<<<<<< HEAD "format": "short", "label": null, "logBase": 1, @@ -622,6 +640,9 @@ "show": true }, { +======= + "decimals": null, +>>>>>>> 08be6d5d0a (metrics: migrate TiFlash Summary dashboard generation to grafanalib (#11027)) "format": "short", "label": null, "logBase": 1, @@ -632,106 +653,113 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "TiFlash CPU usage calculated with process CPU running seconds.", + "decimals": 3, + "description": "The available capacity size per TiFlash instance", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 5, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, - "grid": {}, "gridPos": { "h": 8, - "w": 12, - "x": 0, - "y": 17 + "w": 8, + "x": 8, + "y": 0 }, - "hiddenSeries": false, - "id": 51, + "height": null, + "hideTimeOverride": false, + "id": 3, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": true, + "hideEmpty": true, + "hideZero": true, + "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "max", + "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, - "linewidth": 1, + "linewidth": 0, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/limit/", - "color": "#F2495C", - "hideTooltip": true, - "legend": false, - "linewidth": 2 - } - ], - "spaceLength": 10, - "stack": false, + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, + "stack": true, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "rate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoreSizeAvailable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A", - "step": 40 - }, - { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", - "interval": "", - "intervalFactor": 1, - "legendFormat": "limit-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoreSizeAvailable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "CPU Usage", + "title": "Available size", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -739,8 +767,8 @@ }, "yaxes": [ { - "decimals": 1, - "format": "percentunit", + "decimals": null, + "format": "bytes", "label": null, "logBase": 1, "max": null, @@ -748,6 +776,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -758,179 +787,113 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "The memory usage per TiFlash instance", + "decimals": 3, + "description": "The capacity size per TiFlash instance", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 5, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, - "w": 12, - "x": 12, - "y": 17 + "w": 8, + "x": 16, + "y": 0 }, - "hiddenSeries": false, - "id": 10, + "height": null, + "hideTimeOverride": false, + "id": 4, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": true, + "hideEmpty": true, + "hideZero": true, + "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "max", + "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, - "linewidth": 1, + "linewidth": 0, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/limit/", - "color": "#F2495C", - "hideTooltip": true, - "legend": false, - "linewidth": 2 - } - ], - "spaceLength": 10, - "stack": false, + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, + "stack": true, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_proxy_process_resident_memory_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoreSizeCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "K" - }, - { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_MemoryCapacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", - "hide": false, - "legendFormat": "limit-{{instance}}", - "refId": "L" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_retained{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "retained", - "refId": "A" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_mapped{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "mapped", - "refId": "B" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_resident{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "resident", - "refId": "C" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_allocated{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "allocated", - "refId": "D" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_active{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "active", - "refId": "E" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_metadata_thp{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "metadata_thp", - "refId": "F" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_jemalloc_metadata{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "metadata", - "refId": "G" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_mimalloc_current_rss{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "mimalloc_rss", - "refId": "H" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_mimalloc_current_commit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "mimalloc_commit", - "refId": "I" - }, - { - "expr": "sum(tiflash_system_asynchronous_metric_mmap_alive{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", - "format": "time_series", - "hide": true, - "intervalFactor": 1, - "legendFormat": "mmap", - "refId": "J" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoreSizeCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "Memory", + "title": "Capacity size", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -938,6 +901,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -946,6 +910,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -956,43 +921,56 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": "TiFlash uptime since last restart", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 25 + "y": 8 }, - "hiddenSeries": false, - "id": 181, + "height": null, + "hideTimeOverride": false, + "id": 5, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": true, + "hideEmpty": true, + "hideZero": true, + "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "max", + "sort": "current", "sortDesc": true, "total": false, "values": true @@ -1000,46 +978,55 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (irate(tiflash_proxy_threads_io_bytes_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_Uptime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_Uptime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "IO Throughput", + "title": "Uptime", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1047,15 +1034,16 @@ }, "yaxes": [ { - "decimals": 0, - "format": "bytes", + "decimals": null, + "format": "dtdurations", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1066,46 +1054,57 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, - "description": "", + "decimals": 0, + "description": "The number of Regions on each TiFlash instance", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 25 + "y": 8 }, - "hiddenSeries": false, - "id": 366, + "height": null, + "hideTimeOverride": false, + "id": 6, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, - "max": false, + "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -1113,45 +1112,70 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null as zero", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_s3_store_summary_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance, store_id,type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_proxy_tikv_raftstore_region_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",type=\"region\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "store-{{store_id}}-{{type}}", - "refId": "A", - "step": 10 + "legendFormat": "{{instance}}", + "metric": "", + "query": "sum((\n tiflash_proxy_tikv_raftstore_region_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",type=\"region\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_proxy_tikv_raftstore_hibernated_peer_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, state) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{state}}", + "metric": "", + "query": "sum((\n tiflash_proxy_tikv_raftstore_hibernated_peer_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, state) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "Remote Store Summary (Disagg arch)", + "title": "Region", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1159,77 +1183,75 @@ }, "yaxes": [ { - "format": "bytes", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } - } - ], - "repeat": null, - "title": "Server", - "type": "row" - }, - { - "collapsed": true, - "datasource": null, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 1 - }, - "id": 139, - "panels": [ + }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "Involved when importing data.", + "description": "TiFlash CPU usage calculated with process CPU running seconds.", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 34 + "y": 16 }, - "hiddenSeries": false, - "id": 141, + "height": null, + "hideTimeOverride": false, + "id": 7, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1238,47 +1260,78 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "seriesOverrides": [], - "spaceLength": 10, + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [ + { + "alias": "/limit/", + "color": "#F2495C", + "hideTooltip": true, + "legend": false, + "linewidth": 2 + } + ], + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"sst_importer.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "rate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[$__rate_interval])", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "rate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[$__rate_interval])", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_LogicalCPUCores\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "limit-{{instance}}", + "metric": "", + "query": "sum((\n tiflash_system_current_metric_LogicalCPUCores\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "SST Import Service", + "title": "CPU Usage", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1295,6 +1348,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1305,43 +1359,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "Involved when importing data.", + "description": "The memory usage per TiFlash instance", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 34 + "y": 16 }, - "hiddenSeries": false, - "id": 154, + "height": null, + "hideTimeOverride": false, + "id": 8, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1350,128 +1416,927 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { - "alias": "Limit", + "alias": "/limit/", "color": "#F2495C", "hideTooltip": true, "legend": false, - "linewidth": 2, - "nullPointMode": "connected" + "linewidth": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"apply_low_.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_proxy_process_resident_memory_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", - "intervalFactor": 2, + "intervalFactor": 1, "legendFormat": "{{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum((\n tiflash_proxy_process_resident_memory_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"apply_low_.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_MemoryCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "Limit", - "refId": "B" - } - ], - "thresholds": [], - "timeFrom": null, - "timeRegions": [], - "timeShift": null, - "title": "SST Apply", - "tooltip": { - "msResolution": false, - "shared": true, - "sort": 0, - "value_type": "individual" - }, - "type": "graph", - "xaxis": { - "buckets": null, - "mode": "time", - "name": null, - "show": true, - "values": [] - }, - "yaxes": [ + "legendFormat": "limit-{{instance}}", + "metric": "", + "query": "sum((\n tiflash_system_current_metric_MemoryCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + }, { - "decimals": 1, - "format": "percentunit", - "label": null, - "logBase": 1, - "max": null, - "min": "0", - "show": true + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_retained\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "retained", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_retained\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "format": "short", - "label": null, - "logBase": 1, - "max": null, - "min": null, - "show": false + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_mapped\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "mapped", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_mapped\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_resident\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "resident", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_resident\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_allocated\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "allocated", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_allocated\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_active\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "active", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_active\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_metadata_thp\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "metadata_thp", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_metadata_thp\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_jemalloc_metadata\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "metadata", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_jemalloc_metadata\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_mimalloc_current_rss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "mimalloc_rss", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_mimalloc_current_rss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_mimalloc_current_commit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "mimalloc_commit", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_mimalloc_current_commit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_mmap_alive\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "mmap", + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_mmap_alive\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Memory", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": null, + "format": "bytes", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true + }, + { + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false + } + ], + "yaxis": { + "align": false, + "alignLevel": 0 + } + }, + { + "aliasColors": {}, + "bars": false, + "cacheTimeout": null, + "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 24 + }, + "height": null, + "hideTimeOverride": false, + "id": 9, + "interval": null, + "isNew": true, + "legend": { + "alignAsTable": true, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, + "min": false, + "rightSide": true, + "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, + "total": false, + "values": true + }, + "lines": true, + "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null", + "options": { + "alertThreshold": true, + "dataLinks": [] + }, + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, + "stack": false, + "steppedLine": false, + "targets": [ + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(irate(\n tiflash_proxy_threads_io_bytes_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 1, + "legendFormat": "{{instance}}", + "metric": "", + "query": "sum(irate(\n tiflash_proxy_threads_io_bytes_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "IO Throughput", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": 0, + "format": "bytes", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true + }, + { + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false + } + ], + "yaxis": { + "align": false, + "alignLevel": 0 + } + }, + { + "aliasColors": {}, + "bars": false, + "cacheTimeout": null, + "datasource": "${DS_TEST-CLUSTER}", + "decimals": 1, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 24 + }, + "height": null, + "hideTimeOverride": false, + "id": 10, + "interval": null, + "isNew": true, + "legend": { + "alignAsTable": true, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": false, + "min": false, + "rightSide": true, + "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, + "total": false, + "values": true + }, + "lines": true, + "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", + "options": { + "alertThreshold": true, + "dataLinks": [] + }, + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, + "stack": false, + "steppedLine": false, + "targets": [ + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_s3_store_summary_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, store_id, type) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "store-{{store_id}}-{{type}}", + "metric": "", + "query": "sum((\n tiflash_storage_s3_store_summary_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, store_id, type) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Remote Store Summary (Disagg arch)", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": null, + "format": "bytes", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true + }, + { + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false + } + ], + "yaxis": { + "align": false, + "alignLevel": 0 + } + } + ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, + "title": "Server", + "transformations": [], + "transparent": false, + "type": "row" + }, + { + "cacheTimeout": null, + "collapsed": true, + "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 0 + }, + "height": null, + "hideTimeOverride": false, + "id": 11, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, + "panels": [ + { + "aliasColors": {}, + "bars": false, + "cacheTimeout": null, + "datasource": "${DS_TEST-CLUSTER}", + "description": "Involved when importing data.", + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, + "height": null, + "hideTimeOverride": false, + "id": 12, + "interval": null, + "isNew": true, + "legend": { + "alignAsTable": true, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, + "min": false, + "rightSide": true, + "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, + "total": false, + "values": true + }, + "lines": true, + "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null", + "options": { + "alertThreshold": true, + "dataLinks": [] + }, + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": true, + "legend": false, + "linewidth": 2, + "nullPointMode": "connected" + } + ], + "span": null, + "stack": false, + "steppedLine": false, + "targets": [ + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"sst_importer.*\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}", + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"sst_importer.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"sst_importer.*\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "Limit", + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"sst_importer.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "SST Import Service", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": null, + "format": "percentunit", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true + }, + { + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": "Involved when importing data.", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, + "height": null, + "hideTimeOverride": false, + "id": 13, + "interval": null, + "isNew": true, + "legend": { + "alignAsTable": true, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, + "min": false, + "rightSide": true, + "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, + "total": false, + "values": true + }, + "lines": true, + "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null", + "options": { + "alertThreshold": true, + "dataLinks": [] + }, + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [ + { + "alias": "Limit", + "color": "#F2495C", + "hideTooltip": true, + "legend": false, + "linewidth": 2, + "nullPointMode": "connected" + } + ], + "span": null, + "stack": false, + "steppedLine": false, + "targets": [ + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_low_.*\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}", + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_low_.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_low_.*\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "Limit", + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_low_.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "SST Apply", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": null, + "format": "percentunit", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true + }, + { + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false + } + ], + "yaxis": { + "align": false, + "alignLevel": 0 + } + }, + { + "aliasColors": {}, + "bars": false, + "cacheTimeout": null, + "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 41 + "y": 8 }, - "hiddenSeries": false, - "id": 145, + "height": null, + "hideTimeOverride": false, + "id": 14, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1480,15 +2345,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -1499,46 +2369,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"region_task.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_task.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_task.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"region_task.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_task.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_task.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Region Task", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1546,7 +2425,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -1555,6 +2434,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1565,43 +2445,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 41 + "y": 8 }, - "hiddenSeries": false, - "id": 147, + "height": null, + "hideTimeOverride": false, + "id": 15, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1610,15 +2502,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -1629,46 +2526,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"region_worker.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_worker.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_worker.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"region_worker.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_worker.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"region_worker.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Region Worker", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1676,7 +2582,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -1685,6 +2591,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1695,43 +2602,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 48 + "y": 16 }, - "hiddenSeries": false, - "id": 155, + "height": null, + "hideTimeOverride": false, + "id": 16, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1740,15 +2659,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -1759,46 +2683,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"raftstore_.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"raftstore_.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"raftstore_.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"raftstore_.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"raftstore_.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"raftstore_.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Store", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1806,7 +2739,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -1815,6 +2748,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1825,43 +2759,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 48 + "y": 16 }, - "hiddenSeries": false, - "id": 257, + "height": null, + "hideTimeOverride": false, + "id": 17, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -1870,15 +2816,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -1889,46 +2840,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"apply_.*\", name!~\"apply_low_.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"apply_.*\", name!~\"apply_low_.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"apply_.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Apply Worker", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -1936,7 +2896,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -1945,6 +2905,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -1955,43 +2916,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 55 + "y": 24 }, - "hiddenSeries": false, - "id": 151, + "height": null, + "hideTimeOverride": false, + "id": 18, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2000,15 +2973,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2019,46 +2997,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"bg_\\\\d+\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_\\\\d+\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_\\\\d+\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"bg_\\\\d+\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_\\\\d+\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_\\\\d+\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Storage Background (Small Tasks)", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2066,7 +3053,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2075,6 +3062,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2085,43 +3073,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 55 + "y": 24 }, - "hiddenSeries": false, - "id": 156, + "height": null, + "hideTimeOverride": false, + "id": 19, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2130,15 +3130,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2149,46 +3154,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"bg_block_\\\\d+\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_block_\\\\d+\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_block_\\\\d+\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"bg_block_\\\\d+\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_block_\\\\d+\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"bg_block_\\\\d+\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Storage Background (Large Tasks)", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2196,7 +3210,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2205,6 +3219,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2215,43 +3230,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "Involved when manually compacting the data.", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 62 + "y": 32 }, - "hiddenSeries": false, - "id": 149, + "height": null, + "hideTimeOverride": false, + "id": 20, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2260,15 +3287,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2279,46 +3311,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"m_compact_pool\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"m_compact_pool\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"m_compact_pool\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"m_compact_pool\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"m_compact_pool\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"m_compact_pool\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Manual Compaction", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2326,7 +3367,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2335,6 +3376,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2345,43 +3387,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 62 + "y": 32 }, - "hiddenSeries": false, - "id": 159, + "height": null, + "hideTimeOverride": false, + "id": 21, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2390,15 +3444,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2409,46 +3468,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"async_poller.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"async_poller.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"async_poller.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"async_poller.*\", instance=~\"$tiflash_role\"}) < sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance) or sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"async_poller.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"async_poller.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "GRPC Async Server", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2456,7 +3524,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2465,6 +3533,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2475,43 +3544,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 69 + "y": 40 }, - "hiddenSeries": false, - "id": 161, + "height": null, + "hideTimeOverride": false, + "id": 22, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2520,15 +3601,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2539,46 +3625,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"GRPCComp.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"GRPCComp.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"GRPCComp.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"GRPCComp.*\", instance=~\"$tiflash_role\"}) < sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance) or sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"GRPCComp.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"GRPCComp.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "GRPC Async Client", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2586,7 +3681,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2595,6 +3690,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2605,43 +3701,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 69 + "y": 40 }, - "hiddenSeries": false, - "id": 256, + "height": null, + "hideTimeOverride": false, + "id": 23, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2650,15 +3758,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2669,46 +3782,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"fap_builder.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"fap_builder.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"fap_builder.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"GRPCComp.*\", instance=~\"$tiflash_role\"}) < sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance) or sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"fap_builder.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"fap_builder.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FAP builder", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2716,7 +3838,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2725,6 +3847,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2735,43 +3858,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 76 + "y": 48 }, - "hiddenSeries": false, - "id": 153, + "height": null, + "hideTimeOverride": false, + "id": 24, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2780,15 +3915,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2799,46 +3939,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"snap_sender.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"snap_sender.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"snap_sender.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"snap_sender.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"snap_sender.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"snap_sender.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Snapshot Sender", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2846,7 +3995,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2855,6 +4004,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2865,43 +4015,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 76 + "y": 48 }, - "hiddenSeries": false, - "id": 267, + "height": null, + "hideTimeOverride": false, + "id": 25, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -2910,15 +4072,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -2929,46 +4096,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"segment_sched.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"segment_sched.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"segment_sched.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"segment_sched.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"segment_sched.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"segment_sched.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Segment Scheduler", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -2976,7 +4152,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -2985,6 +4161,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -2995,43 +4172,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 83 + "y": 56 }, - "hiddenSeries": false, - "id": 295, + "height": null, + "hideTimeOverride": false, + "id": 26, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -3040,15 +4229,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -3059,58 +4253,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"LocalIndexPool*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"LocalIndexPool*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "pool-{{instance}}", - "refId": "A", - "step": 40 - }, - { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"LocalIndexPool*\", instance=~\"$tiflash_role\"})", - "hide": true, - "interval": "", - "intervalFactor": 2, - "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"LocalIndexPool*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"LocalIndexSched*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"LocalIndexPool*\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "sched-{{instance}}", - "refId": "C", - "step": 40 + "legendFormat": "Limit", + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"LocalIndexPool*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Local Index Pool", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3118,7 +4309,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -3127,6 +4318,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -3137,43 +4329,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 83 + "y": 56 }, - "hiddenSeries": false, - "id": 268, + "height": null, + "hideTimeOverride": false, + "id": 27, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -3182,15 +4386,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -3201,46 +4410,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\", instance=~\"$proxy_instance\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"SegmentReader.*\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"SegmentReader.*\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\", instance=~\"$proxy_instance\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"SegmentReader.*\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count((\n tiflash_proxy_thread_cpu_seconds_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",name=~\"SegmentReader.*\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Segment Reader", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3248,7 +4466,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -3257,6 +4475,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -3267,55 +4486,98 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Threads CPU", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 2 + "y": 0 }, - "id": 327, + "height": null, + "hideTimeOverride": false, + "id": 28, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 35 + "y": 0 }, - "hiddenSeries": false, - "id": 329, + "height": null, + "hideTimeOverride": false, + "id": 29, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -3323,53 +4585,70 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, - "points": true, + "pointradius": 5, + "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_proxy_threads_state{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}) by (instance, state)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_proxy_threads_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, state) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{state}}", - "refId": "A", - "step": 4 + "metric": "", + "query": "sum((\n tiflash_proxy_threads_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, state) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_proxy_threads_state{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_proxy_threads_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-total", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_proxy_threads_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads state", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3377,6 +4656,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -3385,44 +4665,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 35 + "y": 0 }, - "hiddenSeries": false, - "id": 331, + "height": null, + "hideTimeOverride": false, + "id": 30, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -3436,45 +4733,55 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, - "points": true, + "pointradius": 5, + "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_threads_io_bytes_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[30s])) by (name, io, $additional_groupby) > 1024", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_threads_io_bytes_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (name, io, $additional_groupby) > 1024", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{name}}-{{io}} {{$additional_groupby}}", - "refId": "A", - "step": 4 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_threads_io_bytes_total\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (name, io, $additional_groupby) > 1024", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads IO", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3482,7 +4789,8 @@ }, "yaxes": [ { - "format": "Bps", + "decimals": null, + "format": "binBps", "label": null, "logBase": 1, "max": null, @@ -3490,44 +4798,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 42 + "y": 8 }, - "hiddenSeries": false, - "id": 333, + "height": null, + "hideTimeOverride": false, + "id": 31, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -3541,45 +4866,55 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, - "points": true, + "pointradius": 5, + "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_thread_voluntary_context_switches{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[30s])) by (instance, name) > 200", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_voluntary_context_switches\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, name) > 200", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}} - {{name}}", - "refId": "A", - "step": 4 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_voluntary_context_switches\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, name) > 200", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Thread Voluntary Context Switches", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3587,6 +4922,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -3595,44 +4931,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 42 + "y": 8 }, - "hiddenSeries": false, - "id": 335, + "height": null, + "hideTimeOverride": false, + "id": 32, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -3646,45 +4999,55 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, - "points": true, + "pointradius": 5, + "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_thread_nonvoluntary_context_switches{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[30s])) by (instance, name) > 50", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_thread_nonvoluntary_context_switches\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, name) > 50", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}} - {{name}}", - "refId": "A", - "step": 4 + "metric": "", + "query": "sum(rate(\n tiflash_proxy_thread_nonvoluntary_context_switches\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, name) > 50", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Thread Nonvoluntary Context Switches", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3692,6 +5055,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -3700,63 +5064,108 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Threads", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 3 + "y": 0 }, - "id": 6, + "height": null, + "hideTimeOverride": false, + "id": 33, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 36 + "y": 0 }, - "hiddenSeries": false, - "id": 9, + "height": null, + "hideTimeOverride": false, + "id": 34, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -3765,41 +5174,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_coprocessor_request_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_coprocessor_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", - "intervalFactor": 1, + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_coprocessor_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Request QPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3816,47 +5239,66 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 36 + "y": 0 }, - "hiddenSeries": false, - "id": 2, + "height": null, + "hideTimeOverride": false, + "id": 35, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -3865,41 +5307,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_coprocessor_executor_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_coprocessor_executor_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", - "intervalFactor": 1, + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_coprocessor_executor_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Executor QPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -3907,6 +5363,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -3915,114 +5372,198 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 43 + "y": 8 }, - "hiddenSeries": false, - "id": 11, + "height": null, + "hideTimeOverride": false, + "id": 36, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "999-{{type}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "99-{{type}}", - "refId": "B" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "95-{{type}}", - "refId": "C" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.80, sum(rate(tiflash_coprocessor_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_coprocessor_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_coprocessor_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "80-{{type}}", - "refId": "D" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_coprocessor_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_coprocessor_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Request Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4030,6 +5571,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -4038,88 +5580,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 43 + "y": 8 }, - "hiddenSeries": false, - "id": 12, + "height": null, + "hideTimeOverride": false, + "id": 37, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_coprocessor_request_error{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (reason)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_coprocessor_request_error\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (reason) ", "format": "time_series", - "intervalFactor": 1, + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "{{reason}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_coprocessor_request_error\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (reason) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Error QPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4127,6 +5704,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -4135,109 +5713,198 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 50 + "y": 16 }, - "hiddenSeries": false, - "id": 13, + "height": null, + "hideTimeOverride": false, + "id": 38, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "histogram_quantile(0.999, sum(rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "999-{{type}}", - "refId": "A" + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "99-{{type}}", - "refId": "B" + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "95-{{type}}", - "refId": "C" + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.80, sum(rate(tiflash_coprocessor_request_handle_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "80-{{type}}", - "refId": "D" + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_handle_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_coprocessor_request_handle_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_coprocessor_request_handle_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_coprocessor_request_handle_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_coprocessor_request_handle_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Request Handle Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4245,6 +5912,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -4253,47 +5921,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 50 + "y": 16 }, - "hiddenSeries": false, - "id": 14, + "height": null, + "hideTimeOverride": false, + "id": 39, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -4302,43 +5989,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_coprocessor_response_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_coprocessor_response_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_coprocessor_response_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Response Bytes/Seconds", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4346,6 +6045,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -4354,107 +6054,168 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 57 + "y": 24 }, - "hiddenSeries": false, - "id": 63, + "height": null, + "hideTimeOverride": false, + "id": 40, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "histogram_quantile(0.999, sum(rate(tiflash_coprocessor_request_memory_usage_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "999-{{type}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.99, sum(rate(tiflash_coprocessor_request_memory_usage_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "99-{{type}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_coprocessor_request_memory_usage_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "95-{{type}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.80, sum(rate(tiflash_coprocessor_request_memory_usage_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "80-{{type}}", - "refId": "D" + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_coprocessor_request_memory_usage_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Cop task memory usage", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4462,6 +6223,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -4470,47 +6232,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 57 + "y": 24 }, - "hiddenSeries": false, - "id": 165, + "height": null, + "hideTimeOverride": false, + "id": 41, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -4519,43 +6300,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_exchange_data_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_exchange_data_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_exchange_data_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Exchange Bytes/Seconds", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4563,6 +6356,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -4571,90 +6365,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 64 + "y": 32 }, - "hiddenSeries": false, - "id": 100, + "height": null, + "hideTimeOverride": false, + "id": 42, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_thread_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"rpc.*\", type!~\".*max\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"rpc.*\",type!~\".*max\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"rpc.*\",type!~\".*max\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads of Rpc", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4662,6 +6489,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -4670,88 +6498,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 64 + "y": 32 }, - "hiddenSeries": false, - "id": 77, + "height": null, + "hideTimeOverride": false, + "id": 43, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(tiflash_coprocessor_handling_request_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_coprocessor_handling_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_coprocessor_handling_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Handling Request Number", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4768,90 +6631,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 71 + "y": 40 }, - "hiddenSeries": false, - "id": 102, + "height": null, + "hideTimeOverride": false, + "id": 44, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_thread_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\".*max\", type!~\"rpc.*\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\".*max\",type!~\"rpc.*\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\".*max\",type!~\"rpc.*\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4859,6 +6755,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -4867,90 +6764,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 71 + "y": 40 }, - "hiddenSeries": false, - "id": 101, + "height": null, + "hideTimeOverride": false, + "id": 45, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_thread_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"rpc.*\", type=~\".*max\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"rpc.*\",type=~\".*max\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"rpc.*\",type=~\".*max\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Max Threads of Rpc", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -4958,6 +6888,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -4966,91 +6897,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The MPP query count in TiFlash", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 78 + "y": 48 }, - "hiddenSeries": false, - "id": 157, + "height": null, + "hideTimeOverride": false, + "id": 46, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_mpp_task_manager{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_mpp_task_manager\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_mpp_task_manager\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "MPP Query count", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5058,6 +7021,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -5066,90 +7030,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 78 - }, - "hiddenSeries": false, - "id": 103, + "y": 48 + }, + "height": null, + "hideTimeOverride": false, + "id": 47, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_thread_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\".*max\", type!~\"rpc.*\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*max\",type!~\"rpc.*\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_thread_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*max\",type!~\"rpc.*\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Max Threads", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5157,6 +7154,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -5165,90 +7163,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 85 + "y": 56 }, - "hiddenSeries": false, - "id": 199, + "height": null, + "hideTimeOverride": false, + "id": 48, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_mpp_task_monitor{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_mpp_task_monitor\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_mpp_task_monitor\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Time of the Longest Live MPP Task", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5256,6 +7287,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -5264,47 +7296,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 85 + "y": 56 }, - "hiddenSeries": false, - "id": 166, + "height": null, + "hideTimeOverride": false, + "id": 49, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -5313,43 +7364,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_exchange_queueing_data_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_exchange_queueing_data_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_exchange_queueing_data_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Data size in send and receive queue", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5357,6 +7420,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -5365,90 +7429,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 92 + "y": 64 }, - "hiddenSeries": false, - "id": 297, + "height": null, + "hideTimeOverride": false, + "id": 50, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_network_transmission_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_network_transmission_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_network_transmission_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Network Transmission", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5456,6 +7553,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -5464,48 +7562,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The establish calldata details", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 92 + "y": 64 }, - "hiddenSeries": false, - "id": 299, + "height": null, + "hideTimeOverride": false, + "id": 51, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -5514,43 +7630,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_establish_calldata_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type != \"new_request_calldata\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_establish_calldata_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type != \"new_request_calldata\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_establish_calldata_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type != \"new_request_calldata\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Establish calldata details", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5558,6 +7686,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -5566,106 +7695,165 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Coprocessor", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 4 + "y": 0 }, - "id": 105, + "height": null, + "hideTimeOverride": false, + "id": 52, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "the min_tso of each instance", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 37 + "y": 0 }, - "hiddenSeries": false, - "id": 107, + "height": null, + "hideTimeOverride": false, + "id": 53, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, - "lines": false, + "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 1, "points": true, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"min_tso\"}) by (instance, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"min_tso\"}\n \n)) by (instance, resource_group) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"min_tso\"}\n \n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Min TSO", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5682,128 +7870,183 @@ "show": false }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "estimated thread usage in min-tso scheduler, and the sort/hard limit of estimated thread in scheduler.", + "editable": true, + "error": false, "fieldConfig": { "defaults": { - "unit": "none" - }, - "overrides": [] + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 37 + "y": 0 }, - "hiddenSeries": false, - "id": 109, + "height": null, + "hideTimeOverride": false, + "id": 54, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 1, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"thread_soft_limit\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"thread_soft_limit\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"thread_soft_limit\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"estimated_thread_usage\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"estimated_thread_usage\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"estimated_thread_usage\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"thread_hard_limit\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"thread_hard_limit\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "C" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"thread_hard_limit\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"global_estimated_thread_usage\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"global_estimated_thread_usage\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "D" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"global_estimated_thread_usage\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"group_entry_count\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"group_entry_count\"}\n \n)) by (instance, type) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "queryType": "randomWalk", - "refId": "E" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"group_entry_count\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Estimated Thread Usage and Limit", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5820,52 +8063,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "the count of active/ waiting queries", + "editable": true, + "error": false, "fieldConfig": { "defaults": { - "unit": "none" - }, - "overrides": [] + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 45 + "y": 8 }, - "hiddenSeries": false, - "id": 111, + "height": null, + "hideTimeOverride": false, + "id": 55, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -5873,51 +8130,71 @@ }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 1, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"waiting_queries_count\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"waiting_queries_count\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"waiting_queries_count\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"active_queries_count\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"active_queries_count\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"active_queries_count\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Active and Waiting Queries Count", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -5934,52 +8211,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "the count of active/ waiting tasks", + "editable": true, + "error": false, "fieldConfig": { "defaults": { - "unit": "none" - }, - "overrides": [] + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 45 + "y": 8 }, - "hiddenSeries": false, - "id": 113, + "height": null, + "hideTimeOverride": false, + "id": 56, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -5987,51 +8278,71 @@ }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 1, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"waiting_tasks_count\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"waiting_tasks_count\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"waiting_tasks_count\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"active_tasks_count\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"active_tasks_count\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"active_tasks_count\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Active and Waiting Tasks Count", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6048,93 +8359,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "the usage of estimated threads exceeded the hard limit where errors occur.", + "editable": true, + "error": false, "fieldConfig": { "defaults": { - "unit": "none" - }, - "overrides": [] + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 53 + "y": 16 }, - "hiddenSeries": false, - "id": 117, + "height": null, + "hideTimeOverride": false, + "id": 57, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 1, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_task_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"hard_limit_exceeded_count\"}) by (instance, type, resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"hard_limit_exceeded_count\"}\n \n)) by (instance, type, resource_group) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_task_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"hard_limit_exceeded_count\"}\n \n)) by (instance, type, resource_group) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Hard Limit Exceeded Count", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6144,116 +8485,205 @@ { "decimals": null, "format": "none", - "label": "", + "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, "description": "the time of waiting for schedule", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 53 + "y": 16 }, - "hiddenSeries": false, - "id": 115, + "height": null, + "hideTimeOverride": false, + "id": 58, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": false, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 1, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.80, max(rate(tiflash_task_scheduler_waiting_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,le,resource_group))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_task_scheduler_waiting_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, instance, resource_group, $additional_groupby) / 1000000000)", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{instance}}-{{resource_group}}-80", - "queryType": "randomWalk", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-max", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_task_scheduler_waiting_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, instance, resource_group, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.90, max(rate(tiflash_task_scheduler_waiting_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,le,resource_group))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-9999", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{instance}}-{{resource_group}}-90", - "queryType": "randomWalk", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-999", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(1.00, max(rate(tiflash_task_scheduler_waiting_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,le,resource_group))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-99", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-80", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, instance, resource_group, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, resource_group, $additional_groupby) / sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, resource_group, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{instance}}-{{resource_group}}-100", - "queryType": "randomWalk", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "{{instance}}-{{resource_group}}-avg", + "metric": "", + "query": "(sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, resource_group, $additional_groupby) / sum(rate(\n tiflash_task_scheduler_waiting_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, resource_group, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Waiting Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6263,135 +8693,217 @@ { "decimals": null, "format": "s", - "label": "Time", + "label": null, "logBase": 1, "max": null, - "min": null, + "min": "0", "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Task Scheduler", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 5 + "y": 0 }, - "id": 16, + "height": null, + "hideTimeOverride": false, + "id": 59, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Executed DDL jobs per minute", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 38 + "y": 0 }, - "hiddenSeries": false, - "id": 19, + "height": null, + "hideTimeOverride": false, + "id": 60, + "interval": null, + "isNew": true, "legend": { "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "avg(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "avg(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "total", - "refId": "B" + "metric": "", + "query": "sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type,instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_schema_internal_ddl_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum(increase(\n tiflash_schema_internal_ddl_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_schema_internal_ddl_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_schema_internal_ddl_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "total-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum(increase(\n tiflash_schema_internal_ddl_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Schema Internal DDL OPM", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6408,89 +8920,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Executed DDL apply jobs per minute", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 38 + "y": 0 }, - "hiddenSeries": false, - "id": 18, + "height": null, + "hideTimeOverride": false, + "id": 61, + "interval": null, + "isNew": true, "legend": { "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "avg(increase(tiflash_schema_trigger_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg(increase(tiflash_schema_trigger_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "triggle-by-{{type}}", - "refId": "A" + "metric": "", + "query": "avg(increase(tiflash_schema_trigger_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Schema Apply OPM", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6507,124 +9053,218 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 45 + "y": 8 }, - "hiddenSeries": false, - "id": 20, + "height": null, + "hideTimeOverride": false, + "id": 62, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "hideZero": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/^applying/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "histogram_quantile(0.999, sum(rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "999-{{type}}", - "refId": "A" + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.99, sum(rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", - "intervalFactor": 1, - "legendFormat": "99-{{type}}", - "refId": "B" + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "95-{{type}}", - "refId": "C" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.80, sum(rate(tiflash_schema_apply_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", keyspace!=\"\"}[1m])) by (le, type, keyspace))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "80-{{type}}", - "refId": "D" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_schema_apply_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(tiflash_sync_schema_applying{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"$type\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_schema_apply_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_schema_apply_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", "format": "time_series", - "intervalFactor": 1, + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_schema_apply_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_schema_apply_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_sync_schema_applying\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"$type\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "applying-{{instance}}", - "refId": "E" + "metric": "", + "query": "sum((\n tiflash_sync_schema_applying\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"$type\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Schema Apply Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6632,6 +9272,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -6640,63 +9281,103 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, - "max": "2", - "min": "0", + "max": null, + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "DDL", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 6 + "y": 0 }, - "id": 307, + "height": null, + "hideTimeOverride": false, + "id": 63, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "TiFlash CPU usage calculated with process CPU running seconds.", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 39 + "y": 0 }, - "hiddenSeries": false, - "id": 310, + "height": null, + "hideTimeOverride": false, + "id": 64, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -6710,15 +9391,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/limit/", @@ -6728,44 +9414,55 @@ "linewidth": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "irate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$tiflash_role\"}[1m])", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "irate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$tiflash_role\"}[$__rate_interval])", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "irate(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$tiflash_role\"}[$__rate_interval])", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_LogicalCPUCores{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_LogicalCPUCores\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "limit-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_LogicalCPUCores\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "CPU Usage (irate)", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6782,6 +9479,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -6792,43 +9490,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 39 + "y": 0 }, - "hiddenSeries": false, - "id": 309, + "height": null, + "hideTimeOverride": false, + "id": 65, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -6837,15 +9547,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -6856,46 +9571,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"}) by (instance)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"SegmentReader.*\", instance=~\"$tiflash_role\"}) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Segment Reader", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -6903,7 +9627,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -6912,6 +9636,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -6922,38 +9647,56 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 47 + "y": 8 }, - "hiddenSeries": false, - "id": 316, + "height": null, + "hideTimeOverride": false, + "id": 66, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -6961,43 +9704,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_coprocessor_request_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_coprocessor_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_coprocessor_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Request QPS by instance", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7014,54 +9769,68 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The flow of different kinds of read operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 47 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 314, + "height": null, + "hideTimeOverride": false, + "id": 67, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -7069,64 +9838,85 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "File Descriptor-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_PSMReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "Page-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_PSMBackgroundReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "PageBackGround-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Throughput by instance", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7134,6 +9924,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -7142,103 +9933,143 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The total count of different kinds of commands received", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 55 + "y": 16 }, - "hiddenSeries": false, - "id": 318, + "height": null, + "hideTimeOverride": false, + "id": 68, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/delete_range|ingest/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_DMWriteBlock{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "write block-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_storage_command_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Command OPS By Instance", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7255,6 +10086,7 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -7265,38 +10097,51 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of write by instance", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 55 + "y": 16 }, - "height": "", - "hiddenSeries": false, - "id": 312, + "height": null, + "hideTimeOverride": false, + "id": 69, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -7310,59 +10155,75 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/total/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "write-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"ingest\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "ingest-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Throughput By Instance", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7370,6 +10231,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -7378,106 +10240,165 @@ "show": true }, { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, - "show": false + "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Imbalance read/write", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 7 + "y": 0 }, - "id": 321, + "height": null, + "hideTimeOverride": false, + "id": 70, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 40 + "y": 0 }, - "hiddenSeries": false, - "id": 325, + "height": null, + "hideTimeOverride": false, + "id": 71, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_NumKeyspace{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_NumKeyspace\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "keyspace-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_NumKeyspace\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Number of Keyspaces", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7485,6 +10406,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -7493,7 +10415,8 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -7503,91 +10426,127 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 40 + "y": 0 }, - "hiddenSeries": false, - "id": 324, + "height": null, + "hideTimeOverride": false, + "id": 72, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_NumStorageDeltaMerge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_NumStorageDeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "tables-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_NumStorageDeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_current_metric_NumIStorage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_NumIStorage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "tables-all-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_current_metric_NumIStorage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Number of Physical Tables", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7595,6 +10554,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -7603,7 +10563,8 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -7613,91 +10574,127 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 48 + "y": 8 }, - "hiddenSeries": false, - "id": 319, + "height": null, + "hideTimeOverride": false, + "id": 73, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_NumSegment{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_NumSegment\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "segments-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_NumSegment\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_NumMemTable{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_NumMemTable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "mem_table-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_NumMemTable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Number of Segments", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7705,6 +10702,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -7713,7 +10711,8 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -7723,91 +10722,127 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 48 + "y": 8 }, - "hiddenSeries": false, - "id": 323, + "height": null, + "hideTimeOverride": false, + "id": 74, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_BytesMemTable{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_BytesMemTable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "bytes-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_BytesMemTable\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_BytesMemTableAllocated{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_BytesMemTableAllocated\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "bytes-allocated-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_BytesMemTableAllocated\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Bytes of MemTables", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7815,6 +10850,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -7823,7 +10859,8 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -7833,34 +10870,50 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The memory usage of mark cache and minmax index cache", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 56 + "y": 16 }, - "hiddenSeries": false, - "id": 238, + "height": null, + "hideTimeOverride": false, + "id": 75, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -7874,15 +10927,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/limit/", @@ -7892,48 +10950,70 @@ "linewidth": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_MarkCacheBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_MarkCacheBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "mark_cache_{{instance}}", - "refId": "L" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_MarkCacheBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_MinMaxIndexFiles{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_MinMaxIndexFiles\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "minmax_index_cache_{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_MinMaxIndexFiles\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_RNMVCCIndexCacheBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_RNMVCCIndexCacheBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "rn_mvcc_index_cache_{{instance}}", - "refId": "B" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_RNMVCCIndexCacheBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Mark Cache and Minmax Index Cache Memory Usage", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -7941,6 +11021,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -7949,6 +11030,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -7959,130 +11041,127 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "cache misses or cache hits of mark_cache.\nBased on this infactor, we can check whether mark_cache is large enough", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 56 + "y": 16 }, - "hiddenSeries": false, - "id": 169, + "height": null, + "hideTimeOverride": false, + "id": 76, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_system_profile_event_MarkCacheMisses{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_system_profile_event_MarkCacheMisses\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "mark cache misses", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_system_profile_event_MarkCacheMisses\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_system_profile_event_MarkCacheHits{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_system_profile_event_MarkCacheHits\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "mark cache hits", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_system_profile_event_MarkCacheHits\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Effectiveness of Mark Cache", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, - "transformations": [ - { - "id": "calculateField", - "options": { - "alias": "mark cache count total", - "binary": { - "left": "mark cache misses", - "operator": "+", - "reducer": "sum", - "right": "mark cache hits" - }, - "mode": "binary", - "reduce": { - "reducer": "sum" - } - } - }, - { - "id": "calculateField", - "options": { - "alias": "mark cache effectiveness", - "binary": { - "left": "mark cache hits", - "operator": "/", - "reducer": "sum", - "right": "mark cache count total" - }, - "mode": "binary", - "reduce": { - "reducer": "sum" - } - } - }, - { - "id": "filterFieldsByName", - "options": { - "include": { - "names": [ - "Time", - "mark cache effectiveness" - ] - } - } - } - ], + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8090,6 +11169,7 @@ }, "yaxes": [ { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -8098,7 +11178,8 @@ "show": true }, { - "format": "percent", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -8108,38 +11189,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Information about schema of column file, to learn the memory usage of schema", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 64 + "y": 24 }, - "hiddenSeries": false, - "id": 168, + "height": null, + "hideTimeOverride": false, + "id": 77, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "current", "sortDesc": true, "total": false, @@ -8147,66 +11245,101 @@ }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_shared_block_schemas{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"current_size\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"current_size\"}\n \n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "current_size-{{instance}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"current_size\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_shared_block_schemas{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"hit_count\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"hit_count\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "hit_count_ops-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"hit_count\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_shared_block_schemas{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"still_used_when_evict\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"still_used_when_evict\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "still_used_when_evict-{{instance}}", - "refId": "C" + "metric": "", + "query": "max((\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"still_used_when_evict\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_shared_block_schemas{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"miss_count\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"miss_count\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "miss_count_ops-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_shared_block_schemas\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"miss_count\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Schema of Column File", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8214,6 +11347,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -8222,107 +11356,143 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 64 + "y": 24 }, - "hiddenSeries": false, - "id": 337, + "height": null, + "hideTimeOverride": false, + "id": 78, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/max_snapshot_lifetime/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_SegmentReadTasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SegmentReadTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "read_tasks-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SegmentReadTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "max_snapshot_lifetime-{{instance}}", - "refId": "N" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Snapshots", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8330,6 +11500,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -8338,6 +11509,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -8348,106 +11520,157 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 72 + "y": 32 }, - "hiddenSeries": false, - "id": 289, + "height": null, + "hideTimeOverride": false, + "id": 79, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "rate(tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}[$__interval])", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "B" + "metric": "", + "query": "rate(tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}[$__interval])", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "-rate(tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}[$__interval])", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "-rate(tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}[$__interval])", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_storages_thread_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"alloc_.*\"}\n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-tot", - "refId": "C" + "metric": "", + "query": "((\n tiflash_storages_thread_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"alloc_.*\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "-tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-tot", - "refId": "D" + "metric": "", + "query": "-tiflash_storages_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Memory by thread", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8455,6 +11678,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -8463,117 +11687,168 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 72 + "y": 32 }, - "hiddenSeries": false, - "id": 272, + "height": null, + "hideTimeOverride": false, + "id": 80, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, "hideEmpty": true, "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}[$__interval])", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "B" + "metric": "", + "query": "rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}[$__interval])", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "-rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}[$__interval])", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "-rate(tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}[$__interval])", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"alloc_.*\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_raft_proxy_thread_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"alloc_.*\"}\n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-tot", - "refId": "C" + "metric": "", + "query": "((\n tiflash_raft_proxy_thread_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"alloc_.*\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "-tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-tot", - "refId": "D" + "metric": "", + "query": "-tiflash_raft_proxy_thread_memory_usage{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dealloc_.*\"}", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Memory by thread (proxy)", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8581,6 +11856,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -8589,100 +11865,138 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 80 + "y": 40 }, - "hiddenSeries": false, - "id": 291, + "height": null, + "hideTimeOverride": false, + "id": 81, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_memory_usage_by_class{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_memory_usage_by_class\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "B" + "metric": "", + "query": "((\n tiflash_memory_usage_by_class\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "rate(tiflash_memory_usage_by_class{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__interval])", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}-rate", - "refId": "A" + "metric": "", + "query": "rate(tiflash_memory_usage_by_class{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__interval])", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Memory by class", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8690,6 +12004,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -8698,89 +12013,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 80 + "y": 40 }, - "hiddenSeries": false, - "id": 263, + "height": null, + "hideTimeOverride": false, + "id": 82, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_MemoryTrackingKVStore{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_MemoryTrackingKVStore\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_MemoryTrackingKVStore\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "KVStore memory", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8788,6 +12137,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -8796,79 +12146,130 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Memory trace", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 8 + "y": 0 }, - "id": 377, + "height": null, + "hideTimeOverride": false, + "id": 83, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 9 + "y": 0 }, - "hiddenSeries": false, - "id": 378, + "height": null, + "hideTimeOverride": false, + "id": 84, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/limit/", @@ -8878,70 +12279,100 @@ "linewidth": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_ia_main_queue_capacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_ia_main_queue_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "capacity-main-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_ia_main_queue_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_ia_small_queue_capacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_ia_small_queue_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "capacity-small-{{instance}}", - "refId": "B" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_ia_small_queue_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_ia_manager_segments_memory_capacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_ia_manager_segments_memory_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "capacity-segments-{{instance}}", - "refId": "C" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_ia_manager_segments_memory_capacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_ia_manager_segments_memory_size{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_ia_manager_segments_memory_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "segments-mem-size-{{instance}}", - "refId": "D" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_ia_manager_segments_memory_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_ia_manager_segments_disk_size{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_ia_manager_segments_disk_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "segments-disk-size-{{instance}}", - "refId": "E" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_ia_manager_segments_disk_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "IA usage", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -8949,6 +12380,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -8957,6 +12389,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -8967,38 +12400,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 9 + "y": 0 }, - "hiddenSeries": false, - "id": 379, + "height": null, + "hideTimeOverride": false, + "id": 85, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9007,62 +12457,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "9999 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_ia_manager_segments_memory_wait_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "IA Segments Memory Wait", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9070,6 +12588,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -9078,48 +12597,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 17 + "y": 8 }, - "hiddenSeries": false, - "id": 381, + "height": null, + "hideTimeOverride": false, + "id": 86, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9128,52 +12665,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_ia_remote_read_segment_cache_hit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "cache-hit {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_ia_remote_read_segment_cache_miss{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "cache-miss {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "IA Segment Remote Read Cache", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9190,48 +12745,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 17 + "y": 8 }, - "hiddenSeries": false, - "id": 380, + "height": null, + "hideTimeOverride": false, + "id": 87, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9240,62 +12813,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "9999 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_ia_remote_read_segment_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "IA Segments Remote Read Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9303,6 +12944,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -9311,48 +12953,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 0, - "y": 25 + "y": 16 }, - "hiddenSeries": false, - "id": 382, + "height": null, + "hideTimeOverride": false, + "id": 88, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9361,52 +13021,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_file_cache_hit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_file_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "file-cache-hit {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_file_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_file_cache_miss{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_file_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "file-cache-miss {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_file_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "ColumnarFile Cache", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9423,48 +13101,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 8, - "y": 25 + "y": 16 }, - "hiddenSeries": false, - "id": 383, + "height": null, + "hideTimeOverride": false, + "id": 89, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9473,62 +13169,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "9999 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Columnar Prefetch Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9536,6 +13300,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -9544,48 +13309,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 16, - "y": 25 + "y": 16 }, - "hiddenSeries": false, - "id": 384, + "height": null, + "hideTimeOverride": false, + "id": 90, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9594,62 +13377,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "99 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", - "hide": false, + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "9999 {{$additional_groupby}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", - "hide": false, + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "99 {{$additional_groupby}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_prefetch_cache_hit_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Columnar Prefetch Cache Hit Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9657,6 +13508,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -9665,48 +13517,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 33 + "y": 24 }, - "hiddenSeries": false, - "id": 386, + "height": null, + "hideTimeOverride": false, + "id": 91, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9715,44 +13585,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_retry_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_retry_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "retry {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_retry_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Columnar Fetch Snapshot Retry", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9769,48 +13650,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 33 + "y": 24 }, - "hiddenSeries": false, - "id": 385, + "height": null, + "hideTimeOverride": false, + "id": 92, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9819,62 +13718,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "9999 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_proxy_kv_engine_columnar_fetch_snapshot_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Columnar Fetch Snapshot Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -9882,6 +13849,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -9890,48 +13858,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, - "w": 12, + "w": 8, "x": 0, - "y": 41 + "y": 32 }, - "hiddenSeries": false, - "id": 387, + "height": null, + "hideTimeOverride": false, + "id": 93, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -9940,60 +13926,218 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_meta_cache_hit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "hit {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_hit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_meta_cache_miss{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "miss {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_miss\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_parse\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "parse {{$additional_groupby}}", + "metric": "", + "query": "sum(rate(\n tiflash_proxy_kv_engine_columnar_meta_cache_parse\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Columnar Meta Cache", + "tooltip": { + "msResolution": true, + "shared": true, + "sort": 0, + "value_type": "individual" + }, + "transformations": [], + "transparent": false, + "type": "graph", + "xaxis": { + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "decimals": null, + "format": "ops", + "label": null, + "logBase": 1, + "max": null, + "min": "0", + "show": true }, { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_kv_engine_columnar_meta_cache_parse{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m])) by ($additional_groupby)", + "decimals": null, + "format": "short", + "label": null, + "logBase": 1, + "max": null, + "min": null, + "show": false + } + ], + "yaxis": { + "align": false, + "alignLevel": 0 + } + }, + { + "aliasColors": {}, + "bars": false, + "cacheTimeout": null, + "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 32 + }, + "height": null, + "hideTimeOverride": false, + "id": 94, + "interval": null, + "isNew": true, + "legend": { + "alignAsTable": true, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, + "min": false, + "rightSide": true, + "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, + "total": false, + "values": true + }, + "lines": true, + "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", + "options": { + "alertThreshold": true, + "dataLinks": [] + }, + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, + "stack": false, + "steppedLine": false, + "targets": [ + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_columnar_meta_cache_entries\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", - "legendFormat": "parse {{$additional_groupby}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "entries-{{instance}}", + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_columnar_meta_cache_entries\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "Columnar Meta Cache", + "title": "Columnar Meta Cache Entries", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10002,7 +14146,7 @@ "yaxes": [ { "decimals": null, - "format": "ops", + "format": "short", "label": null, "logBase": 1, "max": null, @@ -10010,106 +14154,123 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, - "w": 12, - "x": 12, - "y": 41 + "w": 8, + "x": 16, + "y": 32 }, - "hiddenSeries": false, - "id": 389, + "height": null, + "hideTimeOverride": false, + "id": 95, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "seriesOverrides": [ - { - "alias": "/entries/", - "yaxis": 2 - } - ], - "spaceLength": 10, + "repeat": null, + "repeatDirection": null, + "seriesOverrides": [], + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_columnar_meta_cache_entries{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_kv_engine_columnar_meta_cache_weighted_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "entries-{{instance}}", - "refId": "A" - }, - { - "exemplar": true, - "expr": "tiflash_proxy_kv_engine_columnar_meta_cache_weighted_size{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", - "hide": false, - "interval": "", + "intervalFactor": 2, "legendFormat": "weighted_size-{{instance}}", - "refId": "B" + "metric": "", + "query": "((\n tiflash_proxy_kv_engine_columnar_meta_cache_weighted_size\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, - "title": "Columnar Meta Cache Gauge", + "title": "Columnar Meta Cache Weighted Size", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10117,6 +14278,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -10125,118 +14287,185 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Columnar Storage", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 9 + "y": 0 }, - "id": 25, + "height": null, + "hideTimeOverride": false, + "id": 96, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The total count of different kinds of commands received", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 41 + "y": 0 }, - "hiddenSeries": false, - "id": 41, + "height": null, + "hideTimeOverride": false, + "id": 97, + "interval": null, + "isNew": true, "legend": { "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/delete_range|ingest/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(increase(tiflash_storage_command_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_DMWriteBlock{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "write block", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Command OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10253,6 +14482,7 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -10263,138 +14493,192 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, "cacheTimeout": null, - "dashLength": 10, - "dashes": false, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 41 + "y": 0 }, - "hiddenSeries": false, - "id": 38, + "height": null, + "hideTimeOverride": false, + "id": 98, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/fs|write/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (\ntiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"} +\ntiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}\n)\n/\nsum by (instance) (\ntiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"} + tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"} ) / sum by (instance) ( tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"} )", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "amp-total-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum by (instance) ( tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"} + tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"} ) / sum by (instance) ( tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"} )", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (\nrate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) +\nrate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m])\n)\n/\nsum by (instance) (\nrate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m])\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m]) )", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "amp-5min-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m]) )", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (\nrate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m]) +\nrate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m])\n)\n/\nsum by (instance) (\nrate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[10m])\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[10m]) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "amp-10min-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[10m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[10m]) )", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (\nrate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m]) +\nrate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m])\n)\n/\nsum by (instance) (\nrate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[30m])\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[30m]) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "amp-30min-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[30m]) ) / sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[30m]) )", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (\nrate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) +\nrate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m])\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "fs-5min-{{instance}}", - "refId": "E" + "metric": "", + "query": "sum by (instance) ( rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) + rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) )", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (instance) (\nrate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m])\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m]) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "write-5min-{{instance}}", - "refId": "F" + "metric": "", + "query": "sum by (instance) ( rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[5m]) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Amplification", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10411,6 +14695,7 @@ "show": true }, { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -10421,38 +14706,51 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of (maybe foreground) tasks of storage in bytes", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 49 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 258, + "height": null, + "hideTimeOverride": false, + "id": 99, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -10466,51 +14764,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/total/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_subtask_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_subtask_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_subtask_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "SubTasks Write Throughput (bytes)", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10518,6 +14825,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -10526,48 +14834,62 @@ "show": true }, { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, - "show": false + "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of (maybe foreground) tasks of storage in rows", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 49 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 259, + "height": null, + "hideTimeOverride": false, + "id": 100, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -10581,51 +14903,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/total/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_subtask_throughput_rows{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_subtask_throughput_rows\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_subtask_throughput_rows\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "SubTasks Write Throughput (rows)", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10633,6 +14964,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -10641,93 +14973,123 @@ "show": true }, { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, - "show": false + "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Total number of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 5, "w": 12, "x": 0, - "y": 57 + "y": 16 }, - "hiddenSeries": false, - "id": 39, + "height": null, + "hideTimeOverride": false, + "id": 101, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, - "steppedLine": true, + "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_subtask_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Small Internal Tasks OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10735,7 +15097,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "ops", "label": null, "logBase": 1, @@ -10744,110 +15106,198 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 5, "w": 12, "x": 12, - "y": 57 + "y": 16 }, - "hiddenSeries": false, - "id": 42, + "height": null, + "hideTimeOverride": false, + "id": 102, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": null, - "sortDesc": null, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": false, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le,type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "max-{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "9999-{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99-{{type}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Small Internal Tasks Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10855,7 +15305,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -10864,93 +15314,123 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Total number of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 5, "w": 12, "x": 0, - "y": 62 + "y": 21 }, - "hiddenSeries": false, - "id": 130, + "height": null, + "hideTimeOverride": false, + "id": 103, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, - "steppedLine": true, + "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_subtask_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Large Internal Tasks OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -10958,7 +15438,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "ops", "label": null, "logBase": 1, @@ -10967,110 +15447,198 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 5, "w": 12, "x": 12, - "y": 62 + "y": 21 }, - "hiddenSeries": false, - "id": 131, + "height": null, + "hideTimeOverride": false, + "id": 104, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": null, - "sortDesc": null, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le,type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "max-{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "9999-{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"(delta_merge|seg_merge|seg_split).*\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99-{{type}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"(delta_merge|seg_merge|seg_split).*\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Large Internal Tasks Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11078,7 +15646,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -11087,105 +15655,153 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, "cacheTimeout": null, - "dashLength": 10, - "dashes": false, "datasource": "${DS_TEST-CLUSTER}", "description": "The current processing number of segments' background management", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 67 + "y": 26 }, - "hiddenSeries": false, - "id": 67, + "height": null, + "hideTimeOverride": false, + "id": 105, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "avg(tiflash_system_current_metric_DT_DeltaMerge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_DT_DeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "delta_merge-{{instance}}", - "refId": "A" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_DT_DeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "avg(tiflash_system_current_metric_DT_SegmentSplit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_DT_SegmentSplit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "seg_split-{{instance}}", - "refId": "B" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_DT_SegmentSplit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "avg(tiflash_system_current_metric_DT_SegmentMerge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_DT_SegmentMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "seg_merge-{{instance}}", - "refId": "C" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_DT_SegmentMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Current Data Management Tasks", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11202,46 +15818,61 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of currently opened file descriptors.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 8, "x": 0, - "y": 74 + "y": 34 }, - "hiddenSeries": false, - "id": 50, + "height": null, + "hideTimeOverride": false, + "id": 106, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -11255,72 +15886,100 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "tiflash_proxy_process_open_fds{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", job=~\".*tiflash\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_proxy_process_open_fds\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_proxy_process_open_fds\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\",job=~\".*tiflash\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_OpenFileForWrite{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_OpenFileForWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "W-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_OpenFileForWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_OpenFileForRead{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_OpenFileForRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "R-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_OpenFileForRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_OpenFileForReadWrite{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_OpenFileForReadWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "RW-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_OpenFileForReadWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Opened File Count", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11328,6 +15987,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -11336,6 +15996,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -11346,30 +16007,44 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of open file descriptors action.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 8, "x": 8, - "y": 74 + "y": 34 }, - "hiddenSeries": false, - "id": 22, + "height": null, + "hideTimeOverride": false, + "id": 107, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, @@ -11389,52 +16064,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_FileOpen{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_FileOpen\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "Open-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_FileOpen\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_FileOpenFailed{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_FileOpenFailed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "OpenFail-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_FileOpenFailed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "File Open OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11442,6 +16135,7 @@ }, "yaxes": [ { + "decimals": null, "format": "ops", "label": null, "logBase": 1, @@ -11450,6 +16144,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -11460,34 +16155,50 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "OPS and duration of fsync operations.\n(Only counting storage engine of TiFlash by now. Not including TiFlash-Proxy)", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 8, "x": 16, - "y": 74 + "y": 34 }, - "hiddenSeries": false, - "id": 52, + "height": null, + "hideTimeOverride": false, + "id": 108, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -11501,56 +16212,75 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/max-fsync/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_FileFSync{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_FileFSync\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "ops-fsync-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_FileFSync\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_system_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"fsync\"}[$__rate_interval]))) by (le, instance) / 1000000000)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "max-fsync-{{instance}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_system_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"fsync\"}[$__rate_interval]))) by (le, instance) / 1000000000)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FSync Status", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11567,6 +16297,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -11577,96 +16308,142 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of different kinds of read operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 81 + "y": 42 }, - "hiddenSeries": false, - "id": 46, + "height": null, + "hideTimeOverride": false, + "id": 109, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_PSMWriteIOCalls{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMWriteIOCalls\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "Page", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMWriteIOCalls\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMWritePages{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMWritePages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "PageFile", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMWritePages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWrite{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_WriteBufferFromFileDescriptorWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "File Descriptor", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_WriteBufferFromFileDescriptorWrite\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Disk Write OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11683,106 +16460,153 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of different kinds of read operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 81 + "y": 42 }, - "hiddenSeries": false, - "id": 47, + "height": null, + "hideTimeOverride": false, + "id": 110, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_PSMReadIOCalls{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMReadIOCalls\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "Page", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMReadIOCalls\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMReadPages{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMReadPages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "PageFile", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMReadPages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_ReadBufferFromFileDescriptorRead{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "File Descriptor", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Disk Read OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11799,54 +16623,68 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The flow of different kinds of write operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 88 + "y": 50 }, - "height": "", - "hiddenSeries": false, - "id": 60, + "height": null, + "hideTimeOverride": false, + "id": 111, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -11854,58 +16692,85 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "File Descriptor", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_WriteBufferFromFileDescriptorWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "Page", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMBackgroundWriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "PageBackGround", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundWriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write flow", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -11913,6 +16778,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -11921,54 +16787,68 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The flow of different kinds of read operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 88 + "y": 50 }, - "height": "", - "hiddenSeries": false, - "id": 59, + "height": null, + "hideTimeOverride": false, + "id": 112, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -11976,58 +16856,85 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "File Descriptor", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_ReadBufferFromFileDescriptorReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "Page", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_PSMBackgroundReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "PageBackGround", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_PSMBackgroundReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read flow", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12035,6 +16942,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -12043,97 +16951,138 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The compression ratio of different compression algorithm", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 96 + "y": 58 }, - "hiddenSeries": false, - "id": 294, + "height": null, + "hideTimeOverride": false, + "id": 113, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_uncompressed_bytes\"}[1m]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_compressed_bytes\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_uncompressed_bytes\"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_compressed_bytes\"}[$__rate_interval]))", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "lz4", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_uncompressed_bytes\"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lz4_compressed_bytes\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_uncompressed_bytes\"}[1m]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_compressed_bytes\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_uncompressed_bytes\"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_compressed_bytes\"}[$__rate_interval]))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "lightweight", - "refId": "B" + "metric": "", + "query": "sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_uncompressed_bytes\"}[$__rate_interval]))/sum(rate(tiflash_storage_pack_compression_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"lightweight_compressed_bytes\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Compression Ratio", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12141,6 +17090,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -12149,90 +17099,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The count of the compression algorithm used by each data part", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 96 + "y": 58 }, - "hiddenSeries": false, - "id": 293, + "height": null, + "hideTimeOverride": false, + "id": 114, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": true, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_pack_compression_algorithm_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_pack_compression_algorithm_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_pack_compression_algorithm_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Compression Algorithm Count", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12240,6 +17223,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -12248,106 +17232,165 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Storage", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 10 + "y": 0 }, - "id": 351, + "height": null, + "hideTimeOverride": false, + "id": 115, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Total number of storage engine read tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 42 + "y": 0 }, - "hiddenSeries": false, - "id": 40, + "height": null, + "hideTimeOverride": false, + "id": 116, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, - "avg": true, + "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_storage_read_tasks_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_read_tasks_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_read_tasks_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Tasks OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12357,186 +17400,285 @@ { "decimals": null, "format": "ops", - "label": "", + "label": null, "logBase": 1, "max": null, "min": "0", "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 42 + "y": 0 }, - "hiddenSeries": false, - "id": 88, + "height": null, + "hideTimeOverride": false, + "id": 117, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/max_snapshot_lifetime/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_current_metric_DT_SegmentReadTasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SegmentReadTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "read_tasks-{{instance}}", - "refId": "I" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SegmentReadTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_current_metric_PSMVCCSnapshotsList{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_PSMVCCSnapshotsList\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "snapshot_list-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_current_metric_PSMVCCSnapshotsList\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_PSMVCCNumSnapshots{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", - "format": "heatmap", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_PSMVCCNumSnapshots\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "num_snapshot-{{instance}}", - "refId": "B" + "metric": "", + "query": "((\n tiflash_system_current_metric_PSMVCCNumSnapshots\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfRead{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "read-{{instance}}", - "refId": "C" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfRead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfReadRaw{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfReadRaw\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "read_raw-{{instance}}", - "refId": "D" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfReadRaw\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfDeltaMerge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfDeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "delta_merge-{{instance}}", - "refId": "E" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfDeltaMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfDeltaCompact{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfDeltaCompact\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "delta_compact-{{instance}}", - "refId": "J" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfDeltaCompact\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfSegmentMerge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfSegmentMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "seg_merge-{{instance}}", - "refId": "F" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfSegmentMerge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfSegmentSplit{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfSegmentSplit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "seg_split-{{instance}}", - "refId": "G" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfSegmentSplit\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_current_metric_DT_SnapshotOfPlaceIndex{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_current_metric_DT_SnapshotOfPlaceIndex\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "place_index-{{instance}}", - "refId": "H" + "metric": "", + "query": "((\n tiflash_system_current_metric_DT_SnapshotOfPlaceIndex\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "max_snapshot_lifetime-{{instance}}", - "refId": "K" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_MaxDTDeltaOldestSnapshotLifetime\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Snapshots", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12544,6 +17686,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -12552,6 +17695,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -12562,109 +17706,187 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 50 + "y": 8 }, - "hiddenSeries": false, - "id": 292, + "height": null, + "hideTimeOverride": false, + "id": 118, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, - "avg": true, - "current": false, - "max": false, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "999-{{type}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "99-{{type}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "95-{{type}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.80, sum(rate(tiflash_read_thread_internal_us_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "80-{{type}}", - "refId": "D" + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_read_thread_internal_us_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_read_thread_internal_us_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_read_thread_internal_us_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_read_thread_internal_us_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_read_thread_internal_us_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Thread Internal Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12672,7 +17894,8 @@ }, "yaxes": [ { - "format": "µs", + "decimals": null, + "format": "\u00b5s", "label": null, "logBase": 1, "max": null, @@ -12680,6 +17903,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -12690,87 +17914,117 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The information of read thread scheduling.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 50 + "y": 8 }, - "hiddenSeries": false, - "id": 269, + "height": null, + "hideTimeOverride": false, + "id": 119, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/push_block/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_read_thread_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"ru_exhausted|sche_active_segment_limit|sche_from_cache|sche_new_task|sche_no_pool|sche_no_ru|sche_no_segment|sche_no_slot|push_block_bytes\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_read_thread_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ru_exhausted|sche_active_segment_limit|sche_from_cache|sche_new_task|sche_no_pool|sche_no_ru|sche_no_segment|sche_no_slot|push_block_bytes\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_read_thread_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ru_exhausted|sche_active_segment_limit|sche_from_cache|sche_new_task|sche_no_pool|sche_no_ru|sche_no_segment|sche_no_slot|push_block_bytes\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Thread Scheduling", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12787,6 +18041,7 @@ "show": true }, { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -12797,105 +18052,151 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The information of data sharing cache hit ratio. Data sharing cache is purpose-built for OLAP workload that can reduce repeated data reads of concurrent table scanning.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 0, - "y": 58 + "y": 16 }, - "hiddenSeries": false, - "id": 132, + "height": null, + "hideTimeOverride": false, + "id": 120, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ + { + "alias": "/cache_hit_ratio/", + "yaxis": 2 + }, { "alias": "/cache_hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_read_thread_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"add_cache_total_bytes_limit\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_read_thread_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"add_cache_total_bytes_limit\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_read_thread_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"add_cache_total_bytes_limit\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit\"}[1m]))/sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit|data_sharing_miss\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit|data_sharing_miss\"}[$__rate_interval])))", + "format": "time_series", "hide": false, "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "data_sharing_cache_hit_ratio", - "refId": "B" + "metric": "", + "query": "(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"data_sharing_hit|data_sharing_miss\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit\"}[1m]))/sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit|extra_column_miss\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit|extra_column_miss\"}[$__rate_interval])))", + "format": "time_series", "hide": true, "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "extra_column_cache_hit_ratio", - "refId": "C" + "metric": "", + "query": "(sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_column_cache_packs{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"extra_column_hit|extra_column_miss\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Data Sharing", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -12912,98 +18213,128 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 8, - "y": 58 + "y": 16 }, - "hiddenSeries": false, - "id": 361, + "height": null, + "hideTimeOverride": false, + "id": 121, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/cache_hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (type,$additional_groupby) (tiflash_storage_read_thread_gauge{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_read_thread_gauge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_storage_read_thread_gauge\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Segment MergedTask", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13020,6 +18351,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -13030,100 +18362,187 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 16, - "y": 58 + "y": 16 }, - "hiddenSeries": false, - "id": 362, + "height": null, + "hideTimeOverride": false, + "id": 122, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, - "avg": true, - "current": false, + "avg": false, + "current": true, + "hideEmpty": true, "hideZero": true, - "max": false, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_read_thread_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type,$additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_read_thread_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_read_thread_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "999-{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_read_thread_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type,$additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "99-{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.80, sum(rate(tiflash_storage_read_thread_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type,$additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "80-{{type}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_read_thread_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_read_thread_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_read_thread_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_read_thread_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_read_thread_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Segment MergedTask Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13131,6 +18550,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -13139,6 +18559,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -13149,110 +18570,187 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "fieldConfig": { - "defaults": {}, - "overrides": [] + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 66 + "y": 24 }, - "hiddenSeries": false, - "id": 301, + "height": null, + "hideTimeOverride": false, + "id": 123, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, - "avg": true, - "current": false, - "max": false, + "avg": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "999-{{type}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "99-{{type}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "95-{{type}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_version_chain_ms_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.80, sum(rate(tiflash_storage_version_chain_ms_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_version_chain_ms_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_version_chain_ms_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "80-{{type}}", - "refId": "D" + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_version_chain_ms_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_version_chain_ms_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "VersionChain", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13260,6 +18758,7 @@ }, "yaxes": [ { + "decimals": null, "format": "ms", "label": null, "logBase": 1, @@ -13268,6 +18767,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -13278,82 +18778,112 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Errors of DeltaIndex", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 66 + "y": 24 }, - "hiddenSeries": false, - "id": 237, + "height": null, + "hideTimeOverride": false, + "id": 124, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_DTDeltaIndexError{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DTDeltaIndexError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "DeltaIndexError-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DTDeltaIndexError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "DeltaIndexError", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13370,83 +18900,131 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Storage Read Pool & Data Sharing", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 11 + "y": 0 }, - "id": 119, + "height": null, + "hideTimeOverride": false, + "id": 125, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The disk usage of PageStorage instances in each TiFlash node", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 43 + "y": 0 }, - "hiddenSeries": false, - "id": 128, + "height": null, + "hideTimeOverride": false, + "id": 126, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/^valid_rate/", @@ -13457,62 +19035,85 @@ "linewidth": 3 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_BlobDiskBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_BlobDiskBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "blob_disk_size-{{instance}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_BlobDiskBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_asynchronous_metric_BlobValidBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_BlobValidBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "blob_valid_size-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_BlobValidBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "sum((tiflash_system_asynchronous_metric_BlobValidBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) / (tiflash_system_asynchronous_metric_BlobDiskBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})) by (instance)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "blob_valid_rate-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((tiflash_system_asynchronous_metric_BlobValidBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) / (tiflash_system_asynchronous_metric_BlobDiskBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_LogDiskBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_LogDiskBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "log_size-{{instance}}", - "refId": "E", - "step": 10 + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_LogDiskBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PageStorage Disk Usage", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13520,6 +19121,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -13528,6 +19130,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -13538,97 +19141,128 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The number of files of PageStorage instances in each TiFlash node", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 43 + "y": 0 }, - "hiddenSeries": false, - "id": 129, + "height": null, + "hideTimeOverride": false, + "id": 127, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_asynchronous_metric_BlobFileNums{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_BlobFileNums\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "blob_file-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_BlobFileNums\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_asynchronous_metric_LogNums{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_asynchronous_metric_LogNums\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "log_file-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_asynchronous_metric_LogNums\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PageStorage File Num", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13636,6 +19270,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -13644,20 +19279,22 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, - "max": "1.1", - "min": "0", - "show": true + "max": null, + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -13667,92 +19304,146 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 51 + "y": 8 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 133, + "id": 128, + "interval": null, "legend": { - "show": true + "show": false }, - "pluginVersion": "6.1.6", + "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_storage_page_write_batch_size_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"v3\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_storage_page_write_batch_size_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"v3\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{le}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "sum(delta(\n tiflash_storage_page_write_batch_size_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"v3\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], + "timeFrom": null, + "timeShift": null, "title": "PageStorage WriteBatch Size", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, - "yBucketBound": "auto", + "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 51 + "y": 8 }, - "hiddenSeries": false, - "id": 158, + "height": null, + "hideTimeOverride": false, + "id": 129, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -13761,78 +19452,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}}-max", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}-max {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}}-999", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "{{type}}-9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{type}}-99", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-99 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{type}}-95", - "refId": "D" + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_page_write_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.80, sum(rate(tiflash_storage_page_write_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_page_write_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_page_write_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{type}}-80", - "refId": "E" + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_page_write_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_page_write_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Page write Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13840,6 +19583,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -13848,50 +19592,67 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 59 + "y": 16 }, - "hiddenSeries": false, - "id": 163, + "height": null, + "hideTimeOverride": false, + "id": 130, + "interval": null, + "isNew": true, "legend": { "alignAsTable": false, "avg": false, "current": false, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, - "sort": "max", + "sideWidth": null, + "sort": "current", "sortDesc": true, "total": false, "values": false @@ -13899,45 +19660,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(increase(tiflash_storage_page_gc_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_storage_page_gc_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(increase(\n tiflash_storage_page_gc_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Page GC Tasks OPM", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -13945,6 +19716,7 @@ }, "yaxes": [ { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -13953,49 +19725,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 59 + "y": 16 }, - "hiddenSeries": false, - "id": 162, + "height": null, + "hideTimeOverride": false, + "id": 131, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -14004,55 +19793,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-max {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_page_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}}-max", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}-9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_page_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}}-99", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-99 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_page_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_page_gc_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_page_gc_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_page_gc_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_page_gc_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Page GC Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14060,6 +19924,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -14068,48 +19933,62 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The number of pages of all TiFlash instance", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 67 + "y": 24 }, - "hiddenSeries": false, - "id": 164, + "height": null, + "hideTimeOverride": false, + "id": 132, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, @@ -14123,56 +20002,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_PagesInMem{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_PagesInMem\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "num_pages-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_PagesInMem\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_system_asynchronous_metric_VersionedEntries{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_system_asynchronous_metric_VersionedEntries\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "num_entries-{{instance}}", - "refId": "B", - "step": 10 + "metric": "", + "query": "((\n tiflash_system_asynchronous_metric_VersionedEntries\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Numer of Pages", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14180,6 +20073,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -14188,46 +20082,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The num of pending writers in PageStorage", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 67 + "y": 24 }, - "hiddenSeries": false, - "id": 231, + "height": null, + "hideTimeOverride": false, + "id": 133, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -14241,44 +20150,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_PSPendingWriterNum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_PSPendingWriterNum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "size-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_PSPendingWriterNum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PageStorage Pending Writers Num", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14286,6 +20206,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -14294,6 +20215,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -14304,44 +20226,57 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 75 + "y": 32 }, - "height": "", - "hiddenSeries": false, - "id": 198, + "height": null, + "hideTimeOverride": false, + "id": 134, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -14349,46 +20284,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_page_data_by_types{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_page_data_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_storage_page_data_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PageStorage stored bytes by type", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14396,6 +20340,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -14404,48 +20349,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The number of tables running under different mode in DeltaTree", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 75 + "y": 32 }, - "hiddenSeries": false, - "id": 123, + "height": null, + "hideTimeOverride": false, + "id": 135, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, @@ -14460,70 +20418,100 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_StoragePoolV2Only{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoragePoolV2Only\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "V2-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoragePoolV2Only\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_StoragePoolV3Only{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoragePoolV3Only\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "V3-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoragePoolV3Only\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_StoragePoolMixMode{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoragePoolMixMode\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "Mix-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoragePoolMixMode\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_StoragePoolUniPS{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_StoragePoolUniPS\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "UniPS-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_StoragePoolUniPS\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Number of Tables", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14531,6 +20519,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -14539,97 +20528,127 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 9, + "h": 8, "w": 24, "x": 0, - "y": 83 + "y": 40 }, - "hiddenSeries": false, - "id": 232, + "height": null, + "hideTimeOverride": false, + "id": 136, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_page_command_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_page_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_page_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PS Command OPS By Instance", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14646,6 +20665,7 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -14656,87 +20676,116 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 9, + "h": 8, "w": 24, "x": 0, - "y": 92 + "y": 48 }, - "hiddenSeries": false, - "id": 345, + "height": null, + "hideTimeOverride": false, + "id": 137, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_page_apply_edit_type{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_page_apply_edit_type\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_page_apply_edit_type\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PS Apply edits OPS By Instance", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14753,6 +20802,7 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -14763,55 +20813,98 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "PageStorage", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 12 + "y": 0 }, - "id": 303, + "height": null, + "hideTimeOverride": false, + "id": 138, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The storage I/O limiter metrics.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 12 + "y": 0 }, - "hiddenSeries": false, - "id": 84, + "height": null, + "hideTimeOverride": false, + "id": 139, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -14819,44 +20912,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_io_limiter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_io_limiter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_io_limiter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "I/O Limiter Throughput", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14873,94 +20977,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Current limit bytes per second of Storage I/O limiter", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 12 + "y": 0 }, - "hiddenSeries": false, - "id": 305, + "height": null, + "hideTimeOverride": false, + "id": 140, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_io_limiter_curr{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_io_limiter_curr\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, instance) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_storage_io_limiter_curr\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "I/O Limiter Threshold", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -14968,7 +21101,7 @@ }, "yaxes": [ { - "decimals": 0, + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -14977,49 +21110,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "I/O Limiter current pending gauge.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 0, - "y": 20 + "y": 8 }, - "hiddenSeries": false, - "id": 86, + "height": null, + "hideTimeOverride": false, + "id": 141, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -15028,80 +21178,120 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/pending/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "avg(tiflash_system_current_metric_RateLimiterPendingWriteRequest{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_RateLimiterPendingWriteRequest\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "other-current-{{instance}}", - "refId": "A" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_RateLimiterPendingWriteRequest\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "avg(tiflash_system_current_metric_IOLimiterPendingBgWriteReq{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_IOLimiterPendingBgWriteReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "bgwrite-current-{{instance}}", - "refId": "B" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_IOLimiterPendingBgWriteReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "avg(tiflash_system_current_metric_IOLimiterPendingFgWriteReq{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_IOLimiterPendingFgWriteReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "fgwrite-current-{{instance}}", - "refId": "C" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_IOLimiterPendingFgWriteReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "avg(tiflash_system_current_metric_IOLimiterPendingBgReadReq{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_IOLimiterPendingBgReadReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "bgread-current-{{instance}}", - "refId": "D" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_IOLimiterPendingBgReadReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "avg(tiflash_system_current_metric_IOLimiterPendingFgReadReq{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((\n tiflash_system_current_metric_IOLimiterPendingFgReadReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "fgread-current-{{instance}}", - "refId": "E" + "metric": "", + "query": "avg((\n tiflash_system_current_metric_IOLimiterPendingFgReadReq\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "I/O Limiter Current Pending Gauge", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15109,7 +21299,7 @@ }, "yaxes": [ { - "decimals": 0, + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -15118,6 +21308,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -15128,39 +21319,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The storage I/O limiter metrics.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 8, - "y": 20 + "y": 8 }, - "hiddenSeries": false, - "id": 266, + "height": null, + "hideTimeOverride": false, + "id": 142, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -15169,49 +21376,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_io_limiter_pending_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_io_limiter_pending_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_io_limiter_pending_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "I/O Limiter Pending OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15228,6 +21446,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -15238,39 +21457,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "I/O Limiter pending duration.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 16, - "y": 20 + "y": 8 }, - "hiddenSeries": false, - "id": 369, + "height": null, + "hideTimeOverride": false, + "id": 143, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -15279,58 +21514,130 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_io_limiter_pending_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_io_limiter_pending_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}-pending-max", - "refId": "F" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_io_limiter_pending_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_io_limiter_pending_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", - "legendFormat": "{{type}}-pending-P999", - "refId": "G" + "intervalFactor": 2, + "legendFormat": "{{type}}-pending-9999", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_io_limiter_pending_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-pending-999", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-pending-99", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-pending-80", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_io_limiter_pending_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_io_limiter_pending_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_io_limiter_pending_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "{{type}}-pending-P99", - "refId": "H" + "intervalFactor": 2, + "legendFormat": "{{type}}-pending-avg", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_io_limiter_pending_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_io_limiter_pending_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "I/O Limiter Pending Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15338,73 +21645,117 @@ }, "yaxes": [ { - "decimals": 0, + "decimals": null, "format": "s", "label": null, "logBase": 1, "max": null, - "min": null, + "min": "0", "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Rate Limiter", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 13 + "y": 0 }, - "id": 64, + "height": null, + "hideTimeOverride": false, + "id": 144, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The stall duration of write and delete range", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 24, "x": 0, - "y": 61 + "y": 0 }, - "hiddenSeries": false, - "id": 62, + "height": null, + "hideTimeOverride": false, + "id": 145, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -15413,58 +21764,135 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "99-delta_merge", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_write_stall_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, instance))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_write_stall_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, instance, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, + "legendFormat": "max-{{type}}-{{instance}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_write_stall_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, instance, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}}-{{instance}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}}-{{instance}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "99-{{type}}-{{instance}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_write_stall_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type, instance) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "max-{{type}}-{{instance}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "80-{{type}}-{{instance}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_write_stall_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, instance, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_write_stall_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance, $additional_groupby) / sum(rate(\n tiflash_storage_write_stall_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}}-{{instance}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_write_stall_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance, $additional_groupby) / sum(rate(\n tiflash_storage_write_stall_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Stall Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15472,6 +21900,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -15480,48 +21909,62 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of write and delta's background management", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 69 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 70, + "height": null, + "hideTimeOverride": false, + "id": 146, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -15535,55 +21978,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write|ingest\"}\n [$__rate_interval]\n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "write+ingest", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write|ingest\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"write|ingest\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"write|ingest\"}\n [$__rate_interval]\n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "ManageDelta", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"write|ingest\"}\n [$__rate_interval]\n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write & Delta Management Throughput", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15591,6 +22049,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -15599,7 +22058,8 @@ "show": true }, { - "format": "bytes", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -15609,38 +22069,51 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of write and delta's background management", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 69 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 137, + "height": null, + "hideTimeOverride": false, + "id": 147, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -15654,53 +22127,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write|ingest\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write|ingest\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "write+ingest", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write|ingest\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!~\"write|ingest\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"write|ingest\"}\n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "ManageDelta", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type!~\"write|ingest\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write & Delta Management Total", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15708,6 +22198,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -15716,7 +22207,8 @@ "show": true }, { - "format": "bytes", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -15726,37 +22218,50 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The throughput of write by instance", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 9, + "h": 8, "w": 24, "x": 0, - "y": 77 + "y": 16 }, - "height": "", - "hiddenSeries": false, - "id": 89, + "height": null, + "hideTimeOverride": false, + "id": 148, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, @@ -15771,59 +22276,75 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/total/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "write-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"ingest\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "ingest-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Throughput By Instance", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15831,6 +22352,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -15839,104 +22361,143 @@ "show": true }, { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, - "show": false + "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The total count of different kinds of commands received", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 9, + "h": 8, "w": 24, "x": 0, - "y": 86 + "y": 24 }, - "hiddenSeries": false, - "id": 90, + "height": null, + "hideTimeOverride": false, + "id": 149, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/delete_range|ingest/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_system_profile_event_DMWriteBlock{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "write block-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMWriteBlock\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_storage_command_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum(increase(\n tiflash_storage_command_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Write Command OPS By Instance", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -15953,6 +22514,7 @@ "show": true }, { + "decimals": null, "format": "opm", "label": null, "logBase": 1, @@ -15963,95 +22525,154 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Storage Write Stall", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 14 + "y": 0 }, - "id": 34, + "height": null, + "hideTimeOverride": false, + "id": 150, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 62 + "y": 0 }, - "hiddenSeries": false, - "id": 167, + "height": null, + "hideTimeOverride": false, + "id": 151, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_stale_read_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_stale_read_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_stale_read_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Stale Read OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16068,88 +22689,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 62 + "y": 0 }, - "hiddenSeries": false, - "id": 35, + "height": null, + "hideTimeOverride": false, + "id": 152, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(rate(tiflash_raft_read_index_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_read_index_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", - "intervalFactor": 1, + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_raft_read_index_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Read Index OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16166,90 +22822,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 69 + "y": 8 }, - "hiddenSeries": false, - "id": 270, + "height": null, + "hideTimeOverride": false, + "id": 153, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_learner_read_failures_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_learner_read_failures_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_raft_learner_read_failures_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Learner Read Failures", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16266,90 +22955,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 69 + "y": 8 }, - "hiddenSeries": false, - "id": 271, + "height": null, + "hideTimeOverride": false, + "id": 154, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_read_index_events_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_read_index_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_raft_read_index_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Index Events", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16366,133 +23088,218 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 76 + "y": 16 }, - "hiddenSeries": false, - "id": 37, + "height": null, + "hideTimeOverride": false, + "id": 155, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/timeout/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "99 {{$additional_groupby}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "95 {{$additional_groupby}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.80, sum(rate(tiflash_raft_wait_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "80 {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_wait_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(increase(tiflash_system_profile_event_RaftWaitIndexTimeout{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_wait_index_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_wait_index_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", "format": "time_series", - "intervalFactor": 1, + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_wait_index_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_wait_index_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(increase(\n tiflash_system_profile_event_RaftWaitIndexTimeout\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-timeout", - "refId": "E" + "metric": "", + "query": "sum(increase(\n tiflash_system_profile_event_RaftWaitIndexTimeout\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Wait Index Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16500,6 +23307,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -16508,117 +23316,198 @@ "show": true }, { - "decimals": 2, + "decimals": null, "format": "opm", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of currently applying snapshots.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 76 + "y": 16 }, - "hiddenSeries": false, - "id": 36, + "height": null, + "hideTimeOverride": false, + "id": 156, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, + "legendFormat": "9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "99 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "95", - "refId": "C" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_read_index_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.80, sum(rate(tiflash_raft_read_index_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_read_index_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_read_index_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", "format": "time_series", "hide": true, - "intervalFactor": 1, - "legendFormat": "80", - "refId": "D" + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_read_index_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_read_index_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Batch Read Index Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16626,6 +23515,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -16634,52 +23524,67 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of applying Raft write logs", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 83 + "y": 24 }, - "hiddenSeries": false, - "id": 82, + "height": null, + "hideTimeOverride": false, + "id": 157, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -16687,93 +23592,190 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": " 100%-{{type}}", - "refId": "D", - "step": 4 + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": " 99%-{{type}}", + "legendFormat": "9999-{{type}} {{$additional_groupby}}", "metric": "", - "refId": "A", - "step": 4 + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[1m])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[1m])) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_apply_write_command_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[$__rate_interval])))", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "avg-write", - "refId": "C", - "step": 4 + "metric": "", + "query": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[1m])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[1m])) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[$__rate_interval])))", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "avg-admin", - "refId": "E", - "step": 4 + "metric": "", + "query": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", cluster_id=~\".*$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[1m])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", cluster_id=~\".*$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-flush_region", - "refId": "B" + "metric": "", + "query": "(sum(rate(tiflash_raft_apply_write_command_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[$__rate_interval])) / sum(rate(tiflash_raft_apply_write_command_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"flush_region\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", cluster_id=~\".*$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[1m])) / sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", cluster_id=~\".*$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[1m]) ) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[$__rate_interval])) / sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-decode", - "refId": "F" + "metric": "", + "query": "(sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[$__rate_interval])) / sum(rate(tiflash_raft_write_data_to_storage_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Apply Raft write logs Duration", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 2, - "value_type": "cumulative" + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -16781,6 +23783,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -16789,20 +23792,22 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -16812,67 +23817,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of decoding Region data into blocks when writing Region data to the storage layer. (Mixed with \"write logs\" and \"apply Snapshot\" operations)", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 90 + "y": 32 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 76, + "id": 158, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_write_data_to_storage_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"decode\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_data_to_storage_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"decode\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_data_to_storage_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"decode\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Region write Duration (decode)", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -16882,67 +23922,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of writing Region data blocks to the storage layer (Mixed with \"write logs\" and \"apply Snapshot\" operations)", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 90 + "y": 32 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 87, + "id": 159, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_write_data_to_storage_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_data_to_storage_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"write\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_data_to_storage_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"write\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Region write Duration (write blocks)", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -16952,69 +24027,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of applying Raft write logs", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 97 + "y": 40 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 81, + "id": 160, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"write\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"write\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"write\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Apply Raft write logs Duration [Heatmap]", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17024,61 +24132,95 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of applying Raft write logs", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 97 + "y": 40 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 93, + "id": 161, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_apply_write_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"admin\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"admin\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_apply_write_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"admin\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Apply Raft admin logs Duration [Heatmap]", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, @@ -17087,74 +24229,106 @@ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 104 + "y": 48 }, - "hiddenSeries": false, - "id": 235, + "height": null, + "hideTimeOverride": false, + "id": 162, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_raft_events_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_raft_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{instance}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}", + "metric": "", + "query": "sum(rate(\n tiflash_raft_raft_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Events QPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -17171,90 +24345,123 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 104 + "y": 48 }, - "hiddenSeries": false, - "id": 241, + "height": null, + "hideTimeOverride": false, + "id": 163, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_raft_frequent_events_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_raft_frequent_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{instance}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}", + "metric": "", + "query": "sum(rate(\n tiflash_raft_raft_frequent_events_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Frequent Events QPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -17271,20 +24478,22 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17294,78 +24503,117 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 111 + "y": 56 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 234, + "id": 164, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_raft_log_gap_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"applied_index\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_raft_log_gap_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"applied_index\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_raft_log_gap_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"applied_index\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_raft_log_gap_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"compact_index\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_raft_log_gap_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"compact_index\"}\n [$__rate_interval]\n)) by (le, type) ", + "format": "heatmap", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "A" + "metric": "", + "query": "sum(delta(\n tiflash_raft_raft_log_gap_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"compact_index\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Raft Log Gap Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "none", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17375,70 +24623,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 111 + "y": 56 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 243, + "id": 165, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_entry_size_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"normal\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_entry_size_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"normal\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_entry_size_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"normal\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Raft Entry Batch Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "none", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17448,80 +24728,117 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 118 + "y": 64 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 244, + "id": 166, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_region_flush_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"unflushed\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_region_flush_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"unflushed\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_region_flush_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"unflushed\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_region_flush_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"flushed\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_region_flush_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"flushed\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "A" + "metric": "", + "query": "sum(delta(\n tiflash_raft_region_flush_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"flushed\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Region Size (by event) Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17531,70 +24848,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 118 + "y": 64 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 264, + "id": 167, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_write_flow_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"big_write_to_region\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"big_write_to_region\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"big_write_to_region\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Big Write To Region Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -17604,64 +24953,95 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 125 + "y": 72 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 261, + "id": 168, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_write_flow_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"write_committed\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write_committed\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"write_committed\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Write Committed Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, @@ -17670,74 +25050,106 @@ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 132 + "y": 80 }, - "hiddenSeries": false, - "id": 240, + "height": null, + "hideTimeOverride": false, + "id": 169, + "interval": null, + "isNew": true, "legend": { - "alignAsTable": false, + "alignAsTable": true, "avg": false, - "current": false, - "max": false, + "current": true, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, - "rightSide": false, + "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, - "values": false + "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_eager_gc_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_eager_gc_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{instance}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}", + "metric": "", + "query": "sum(rate(\n tiflash_raft_eager_gc_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Eager GC OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -17754,52 +25166,67 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of Raft logs eager GC tasks", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 132 + "y": 80 }, - "hiddenSeries": false, - "id": 239, + "height": null, + "hideTimeOverride": false, + "id": 170, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -17807,78 +25234,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_eager_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_eager_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": " 99%-{{type}}", + "legendFormat": "max-{{type}} {{$additional_groupby}}", "metric": "", - "refId": "A", - "step": 4 + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_eager_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_raft_eager_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "95%-{{type}}", - "refId": "B", - "step": 4 + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_eager_gc_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type) / sum(rate(tiflash_raft_eager_gc_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "avg-{{type}}", - "refId": "C", - "step": 4 + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_eager_gc_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_eager_gc_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_eager_gc_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_eager_gc_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": " 100%-{{type}}", - "refId": "D", - "step": 4 + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_eager_gc_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_eager_gc_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft Eager GC Duration", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 2, - "value_type": "cumulative" + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -17886,6 +25365,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -17894,54 +25374,68 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The keys flow of different kinds of Raft operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 139 + "y": 88 }, - "height": "", - "hiddenSeries": false, - "id": 71, + "height": null, + "hideTimeOverride": false, + "id": 171, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -17949,46 +25443,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_process_keys{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_process_keys\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_raft_process_keys\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Keys flow", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -17996,6 +25499,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -18004,54 +25508,68 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 146 + "y": 96 }, - "height": "", - "hiddenSeries": false, - "id": 260, + "height": null, + "hideTimeOverride": false, + "id": 172, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -18059,46 +25577,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_throughput_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_raft_throughput_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Raft throughput", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18106,6 +25633,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -18114,20 +25642,22 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -18137,61 +25667,95 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Latency that TiKV sends raft log to TiFlash.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 153 + "y": 104 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 92, + "id": 173, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Upstream Latency [Heatmap]", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, @@ -18200,36 +25764,50 @@ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Latency that TiKV sends raft log to TiFlash.", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 153 + "y": 104 }, - "hiddenSeries": false, - "id": 91, + "height": null, + "hideTimeOverride": false, + "id": 174, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -18237,70 +25815,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, + "interval": "", "intervalFactor": 2, - "legendFormat": " 100%", + "legendFormat": "max {{$additional_groupby}}", "metric": "", - "refId": "A", - "step": 4 + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, - "legendFormat": " 99%", + "legendFormat": "99 {{$additional_groupby}}", "metric": "", - "refId": "B", - "step": 4 + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "histogram_quantile(0.95, sum(rate(tiflash_raft_upstream_latency_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 2, - "legendFormat": "95%", - "refId": "C", - "step": 4 + "legendFormat": "80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_upstream_latency_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_raft_upstream_latency_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) / sum(rate(tiflash_raft_upstream_latency_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_upstream_latency_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_upstream_latency_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, - "legendFormat": "avg", - "refId": "D", - "step": 4 + "legendFormat": "avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_upstream_latency_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) / sum(rate(\n tiflash_raft_upstream_latency_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by ($additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Upstream Latency", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 0, - "value_type": "cumulative" + "sort": 2, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18308,6 +25946,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -18316,90 +25955,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 160 + "y": 112 }, - "hiddenSeries": false, - "id": 296, + "height": null, + "hideTimeOverride": false, + "id": 175, + "interval": null, + "isNew": true, "legend": { "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_proxy_tikv_server_raft_append_rejects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_proxy_tikv_server_raft_append_rejects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(tiflash_proxy_tikv_server_raft_append_rejects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Log Replication Rejected", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18416,68 +26088,109 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Raft", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 15 + "y": 0 }, - "id": 349, + "height": null, + "hideTimeOverride": false, + "id": 176, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 63 + "y": 0 }, - "hiddenSeries": false, - "id": 242, + "height": null, + "hideTimeOverride": false, + "id": 177, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -18485,45 +26198,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_raft_command_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_command_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", "format": "time_series", + "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "99%-{{type}}", - "refId": "D", - "step": 4 + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_raft_command_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_raft_command_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Heavy Raft Apply Duration", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 2, - "value_type": "cumulative" + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18531,6 +26329,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -18539,118 +26338,168 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "The number of currently applying snapshots.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 24, "x": 0, - "y": 70 + "y": 8 }, - "hiddenSeries": false, - "id": 75, + "height": null, + "hideTimeOverride": false, + "id": 178, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "sum(tiflash_system_current_metric_RaftNumSnapshotsPendingApply{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_RaftNumSnapshotsPendingApply\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "Pending-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_RaftNumSnapshotsPendingApply\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(tiflash_system_current_metric_RaftNumPrehandlingSubTasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_RaftNumPrehandlingSubTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "PrehandleSubtasks-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_RaftNumPrehandlingSubTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(tiflash_system_current_metric_RaftNumParallelPrehandlingTasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_RaftNumParallelPrehandlingTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "ParallelTasks-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_RaftNumParallelPrehandlingTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_RaftNumWaitedParallelPrehandlingTasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_RaftNumWaitedParallelPrehandlingTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "Pending-ParallelTasks-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_RaftNumWaitedParallelPrehandlingTasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Applying snapshots Count", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18658,6 +26507,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -18666,20 +26516,22 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -18689,64 +26541,95 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 77 + "y": 16 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 262, + "id": 179, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_write_flow_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"snapshot_uncommitted\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"snapshot_uncommitted\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"snapshot_uncommitted\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Uncommitted Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, @@ -18755,74 +26638,106 @@ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 77 + "y": 16 }, - "hiddenSeries": false, - "id": 249, + "height": null, + "hideTimeOverride": false, + "id": 180, + "interval": null, + "isNew": true, "legend": { + "alignAsTable": false, "avg": false, "current": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, + "rightSide": false, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_raft_ongoing_snapshot_total_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_raft_ongoing_snapshot_total_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_raft_ongoing_snapshot_total_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Ongoing raft snapshot", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -18830,6 +26745,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -18838,20 +26754,22 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -18861,69 +26779,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 84 + "y": 24 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 250, + "id": 181, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_snapshot_total_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"approx_raft_snapshot\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_snapshot_total_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"approx_raft_snapshot\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_snapshot_total_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"approx_raft_snapshot\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -18933,67 +26884,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of pre-decode when applying region snapshot", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 84 + "y": 24 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 72, + "id": 182, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"snapshot_predecode\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_predecode\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_predecode\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Predecode Duration", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19003,69 +26989,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 91 + "y": 32 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 290, + "id": 183, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_command_throughput_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"prehandle_snapshot\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_command_throughput_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"prehandle_snapshot\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_command_throughput_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"prehandle_snapshot\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Prehandle Throughput Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19075,67 +27094,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of pre-decode when applying region snapshot", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 91 + "y": 32 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 73, + "id": 184, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"snapshot_flush\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_flush\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_flush\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Flush Duration", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19145,70 +27199,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 98 + "y": 40 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 265, + "id": 185, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_raft_write_flow_bytes_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"ingest_uncommitted\"}[1m])) by (le, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest_uncommitted\"}\n [$__rate_interval]\n)) by (le, type) ", "format": "heatmap", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_write_flow_bytes_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"ingest_uncommitted\"}\n [$__rate_interval]\n)) by (le, type) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Ingest Uncommitted Size Heatmap", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "bytes", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19218,67 +27304,102 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, - "min": 0, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of SST to DT in pre-decode when applying region snapshot", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 98 + "y": 40 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 127, + "id": 186, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"snapshot_predecode_sst2dt\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_predecode_sst2dt\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"snapshot_predecode_sst2dt\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Snapshot Predecode SST to DT Duration", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19288,123 +27409,210 @@ "colorScale": "sqrt", "colorScheme": "interpolateSpectral", "exponent": 0.5, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of ingesting SST", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 105 + "y": 48 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 74, + "id": 187, + "interval": null, "legend": { - "show": true + "show": false }, "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "expr": "sum(delta(tiflash_raft_command_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"ingest_sst\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"ingest_sst\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, + "interval": "", "intervalFactor": 2, "legendFormat": "{{le}}", - "refId": "B" + "metric": "", + "query": "sum(delta(\n tiflash_raft_command_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"ingest_sst\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], "timeFrom": null, "timeShift": null, "title": "Ingest SST Duration", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": true + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "s", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Raft Snapshot / IngestSST", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 16 + "y": 0 }, - "id": 95, + "height": null, + "hideTimeOverride": false, + "id": 188, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 64 + "y": 0 }, - "hiddenSeries": false, - "id": 99, + "height": null, + "hideTimeOverride": false, + "id": 189, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/^RS Filter/", @@ -19419,66 +27627,100 @@ "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "expr": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (instance)", "format": "time_series", "hide": false, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "1min-{{instance}}", - "refId": "B" + "metric": "", + "query": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[5m]))) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (instance)", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "5min-{{instance}}", - "refId": "C" + "metric": "", + "query": "avg((rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]) - rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / (rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_DMFileFilterNoFilter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMFileFilterNoFilter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": true, "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "No Filter-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMFileFilterNoFilter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_DMFileFilterAftPKAndPackSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMFileFilterAftPKAndPackSet\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": true, "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "PK Filter-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMFileFilterAftPKAndPackSet\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "expr": "sum(rate(tiflash_system_profile_event_DMFileFilterAftRoughSet{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_DMFileFilterAftRoughSet\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", "format": "time_series", "hide": true, + "instant": false, + "interval": "", "intervalFactor": 1, "legendFormat": "RS Filter-{{instance}}", - "refId": "E" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_DMFileFilterAftRoughSet\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Rough Set Filter Rate", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -19497,7 +27739,7 @@ { "decimals": null, "format": "short", - "label": "", + "label": null, "logBase": 1, "max": null, "min": null, @@ -19506,10 +27748,11 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { + "cacheTimeout": null, "cards": { "cardPadding": null, "cardRound": null @@ -19517,110 +27760,190 @@ "color": { "cardColor": "#b4ff00", "colorScale": "sqrt", - "colorScheme": "interpolateOranges", + "colorScheme": "interpolateSpectral", "exponent": 0.5, + "max": null, + "min": null, "mode": "spectrum" }, "dataFormat": "tsbuckets", "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 64 + "y": 0 }, "heatmap": {}, + "height": null, + "hideTimeOverride": false, "hideZeroBuckets": true, "highlightCards": true, - "id": 97, + "id": 190, + "interval": null, "legend": { "show": false }, - "pluginVersion": "6.1.6", + "links": [], + "maxDataPoints": 512, + "maxPerRow": null, + "minSpan": null, + "options": {}, + "repeat": null, + "repeatDirection": null, "reverseYBuckets": false, + "span": null, "targets": [ { - "exemplar": true, - "expr": "sum(delta(tiflash_storage_rough_set_filter_rate_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(delta(\n tiflash_storage_rough_set_filter_rate_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le) ", "format": "heatmap", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{le}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "sum(delta(\n tiflash_storage_rough_set_filter_rate_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le) ", + "refId": "", + "step": 10, + "target": "" } ], + "timeFrom": null, + "timeShift": null, "title": "Rough Set Filter Rate Histogram", "tooltip": { + "msResolution": true, + "shared": true, "show": true, - "showHistogram": false + "showHistogram": true, + "sort": 0, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "heatmap", "xAxis": { - "show": true + "mode": "time", + "name": null, + "show": true, + "values": [] }, "xBucketNumber": null, "xBucketSize": null, "yAxis": { - "decimals": 0, + "decimals": null, "format": "percent", + "label": null, "logBase": 1, "max": null, "min": null, - "show": true, - "splitFactor": null + "show": true }, - "yBucketBound": "auto", + "yBucketBound": "upper", "yBucketNumber": null, "yBucketSize": null } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Rough Set Filter Rate Histogram", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 17 + "y": 0 }, - "id": 171, + "height": null, + "hideTimeOverride": false, + "id": 191, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "PageStorage Checkpoint Duration", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 33 + "y": 0 }, - "hiddenSeries": false, - "id": 187, + "height": null, + "hideTimeOverride": false, + "id": 192, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -19629,60 +27952,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_checkpoint_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_checkpoint_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "{{type}}-max {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_checkpoint_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_checkpoint_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", - "legendFormat": "{{type}}-999 {{$additional_groupby}}", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "{{type}}-9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_checkpoint_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}-99 {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_checkpoint_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_checkpoint_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_checkpoint_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_checkpoint_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_checkpoint_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Checkpoint Upload Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -19690,6 +28083,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -19698,54 +28092,68 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The flow of checkpoint operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 33 + "y": 0 }, - "height": "", - "hiddenSeries": false, - "id": 174, + "height": null, + "hideTimeOverride": false, + "id": 193, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -19753,55 +28161,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_checkpoint_flow{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"incremental\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_checkpoint_flow\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"incremental\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "incremental {{$additional_groupby}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_checkpoint_flow\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"incremental\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_checkpoint_flow{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"compaction\"}[1m])) by ($additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_checkpoint_flow\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"compaction\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "compaction {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_checkpoint_flow\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"compaction\"}\n [$__rate_interval]\n)) by ($additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Checkpoint Upload flow", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -19809,6 +28232,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -19817,54 +28241,67 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": 1, "description": "The keys of checkpoint operations. All keys are uploaded in the checkpoint. Grouped by key types.", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 41 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 196, + "height": null, + "hideTimeOverride": false, + "id": 194, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -19872,46 +28309,55 @@ "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_checkpoint_keys_by_types{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_checkpoint_keys_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_checkpoint_keys_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Checkpoint Upload keys speed by type (all)", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -19919,6 +28365,7 @@ }, "yaxes": [ { + "decimals": null, "format": "ops", "label": null, "logBase": 1, @@ -19927,54 +28374,68 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The flow of checkpoint operations. Group by key types", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 41 + "y": 8 }, - "height": "", - "hiddenSeries": false, - "id": 197, + "height": null, + "hideTimeOverride": false, + "id": 195, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, - "sort": "current", + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -19982,46 +28443,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", - "repeatedByRow": true, + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_checkpoint_flow_by_types{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_checkpoint_flow_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(\n tiflash_storage_checkpoint_flow_by_types\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Checkpoint Upload flow by type (incremental+compaction)", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20029,6 +28499,7 @@ }, "yaxes": [ { + "decimals": null, "format": "binBps", "label": null, "logBase": 1, @@ -20037,99 +28508,124 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The number of files of owned by each TiFlash node", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 49 + "y": 16 }, - "hiddenSeries": false, - "id": 176, + "height": null, + "hideTimeOverride": false, + "id": 196, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"num_files\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"num_files\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "checkpoint_data-{{instance}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"num_files\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote File Num", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20137,6 +28633,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -20145,68 +28642,89 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, - "max": "1.1", - "min": "0", - "show": true + "max": null, + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "The remote store usage owned by each TiFlash node", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 49 + "y": 16 }, - "hiddenSeries": false, - "id": 175, + "height": null, + "hideTimeOverride": false, + "id": 197, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/^valid_rate/", @@ -20217,51 +28735,70 @@ "linewidth": 3 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"total_size\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"total_size\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "remote_size-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"total_size\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"valid_size\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"valid_size\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "valid_size-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_storage_remote_stats\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"valid_size\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "sum((tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"valid_size\"}) / (tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"total_size\"})) by (instance)", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "valid_rate-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"valid_size\"}) / (tiflash_storage_remote_stats{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"total_size\"})) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Store Usage", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20269,6 +28806,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -20277,6 +28815,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -20287,82 +28826,112 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 57 + "y": 24 }, - "hiddenSeries": false, - "id": 189, + "height": null, + "hideTimeOverride": false, + "id": 198, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, "hideEmpty": true, - "hideZero": false, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_disaggregated_object_lock_request_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_disaggregated_object_lock_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_disaggregated_object_lock_request_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Object Lock Request QPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20379,43 +28948,61 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 57 + "y": 24 }, - "hiddenSeries": false, - "id": 191, + "height": null, + "hideTimeOverride": false, + "id": 199, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, + "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -20428,42 +29015,131 @@ }, "lines": true, "linewidth": 1, - "nullPointMode": "null", + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_disaggregated_object_lock_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_disaggregated_object_lock_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_disaggregated_object_lock_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "99%-{{type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_disaggregated_object_lock_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Object Lock Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20471,6 +29147,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -20479,49 +29156,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 8, "x": 0, - "y": 65 + "y": 32 }, - "hiddenSeries": false, - "id": 365, + "height": null, + "hideTimeOverride": false, + "id": 200, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, @@ -20536,45 +29225,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_s3_store_summary_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance, store_id,type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_s3_store_summary_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, store_id, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "store-{{store_id}}-{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_storage_s3_store_summary_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, store_id, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Store Summary", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20582,6 +29281,7 @@ }, "yaxes": [ { + "decimals": null, "format": "bytes", "label": null, "logBase": 1, @@ -20590,43 +29290,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, - "w": 9, + "w": 8, "x": 8, - "y": 65 + "y": 32 }, - "hiddenSeries": false, - "id": 193, + "height": null, + "hideTimeOverride": false, + "id": 201, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, + "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -20639,15 +29357,21 @@ }, "lines": true, "linewidth": 1, - "nullPointMode": "null", + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/total/", @@ -20662,41 +29386,115 @@ "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_s3_gc_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_gc_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "99%-{{type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_gc_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_s3_gc_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", - "legendFormat": "90%-{{type}} {{$additional_groupby}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_gc_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_s3_gc_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_gc_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_s3_gc_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_gc_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote GC Duration Breakdown", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20704,6 +29502,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -20712,7 +29511,8 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, @@ -20722,39 +29522,50 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, - "w": 7, - "x": 17, - "y": 65 + "w": 8, + "x": 16, + "y": 32 }, - "hiddenSeries": false, - "id": 195, + "height": null, + "hideTimeOverride": false, + "id": 202, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, @@ -20769,45 +29580,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_s3_gc_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_s3_gc_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_storage_s3_gc_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote GC Status", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20815,6 +29636,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -20823,49 +29645,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 73 + "y": 40 }, - "hiddenSeries": false, - "id": 363, + "height": null, + "hideTimeOverride": false, + "id": 203, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, @@ -20880,45 +29714,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_s3_lock_mgr_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_s3_lock_mgr_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{instance}}-{{type}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum((\n tiflash_storage_s3_lock_mgr_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Local Lock Manager status", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -20926,6 +29770,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -20934,92 +29779,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 73 + "y": 40 }, - "hiddenSeries": false, - "id": 364, + "height": null, + "hideTimeOverride": false, + "id": 204, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, "hideEmpty": true, - "hideZero": false, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_s3_lock_mgr_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_s3_lock_mgr_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_s3_lock_mgr_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Local Lock Manager QPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21036,97 +29912,128 @@ "show": true }, { - "format": "none", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 81 + "y": 48 }, - "hiddenSeries": false, - "id": 251, + "height": null, + "hideTimeOverride": false, + "id": 205, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_fap_task_result{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_fap_task_result\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_fap_task_result\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FAP result", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21143,6 +30050,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -21153,87 +30061,117 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 81 + "y": 48 }, - "hiddenSeries": false, - "id": 252, + "height": null, + "hideTimeOverride": false, + "id": 206, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_fap_task_state{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_fap_task_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_fap_task_state\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FAP state", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21250,6 +30188,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -21260,87 +30199,192 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 89 + "y": 56 }, - "hiddenSeries": false, - "id": 254, + "height": null, + "hideTimeOverride": false, + "id": 207, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": false, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(round(1000000000*rate(tiflash_fap_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_fap_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-max {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_fap_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, - "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "{{type}}-9999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-99 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_fap_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_fap_task_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_fap_task_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_fap_task_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_fap_task_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FAP time by stage", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21357,97 +30401,128 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 89 + "y": 56 }, - "hiddenSeries": false, - "id": 253, + "height": null, + "hideTimeOverride": false, + "id": 208, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_fap_nomatch_reason{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_fap_nomatch_reason\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_fap_nomatch_reason\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "FAP no match reason", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21464,6 +30539,7 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -21474,48 +30550,92 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Disaggregated-Write", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 18 + "y": 0 }, - "id": 347, + "height": null, + "hideTimeOverride": false, + "id": 209, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 24, "x": 0, - "y": 34 + "y": 0 }, - "hiddenSeries": false, - "id": 173, + "height": null, + "hideTimeOverride": false, + "id": 210, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, + "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -21528,42 +30648,131 @@ }, "lines": true, "linewidth": 1, - "nullPointMode": "null", + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_disaggregated_breakdown_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_disaggregated_breakdown_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_disaggregated_breakdown_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "99%-{{type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_disaggregated_breakdown_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Read Duration Breakdown", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21571,6 +30780,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -21579,49 +30789,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Remote Cache Operations", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 42 + "y": 8 }, - "hiddenSeries": false, - "id": 185, + "height": null, + "hideTimeOverride": false, + "id": 211, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -21630,15 +30857,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "dtfile_cache_hit_ratio", @@ -21649,50 +30881,70 @@ "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_remote_cache\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_remote_cache\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit\"}[1m]))/sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit|dtfile_miss\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit|dtfile_miss\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "dtfile_cache_hit_ratio", - "refId": "A" + "metric": "", + "query": "(sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"dtfile_hit|dtfile_miss\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit\"}[1m]))/sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit|page_miss\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit|page_miss\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "page_cache_hit_ratio", - "refId": "C" + "metric": "", + "query": "(sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit\"}[$__rate_interval])) / sum(rate(tiflash_storage_remote_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"page_hit|page_miss\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Operations", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21709,49 +30961,66 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Remote Cache Flow", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 42 + "y": 8 }, - "hiddenSeries": false, - "id": 186, + "height": null, + "hideTimeOverride": false, + "id": 212, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -21760,44 +31029,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_remote_cache_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_remote_cache_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Flow", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21814,43 +31094,61 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 50 + "y": 16 }, - "hiddenSeries": false, - "id": 373, + "height": null, + "hideTimeOverride": false, + "id": 213, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -21863,60 +31161,131 @@ }, "lines": true, "linewidth": 1, - "nullPointMode": "null", + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_remote_cache_bg_download_stage_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, stage, file_type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_remote_cache_bg_download_stage_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, stage, file_type, $additional_groupby) / 1000000000)", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "999%-{{stage}}-{{file_type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "B" + "intervalFactor": 2, + "legendFormat": "max-{{stage}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_remote_cache_bg_download_stage_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, stage, file_type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_remote_cache_bg_download_stage_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, stage, file_type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", - "legendFormat": "99%-{{stage}}-{{file_type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "D" + "intervalFactor": 2, + "legendFormat": "9999-{{stage}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "(sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_sum\n {k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) / sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_count\n {k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) )", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{stage}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{stage}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{stage}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, stage, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) / sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-{{stage}}-{{file_type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "(sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) / sum(rate(\n tiflash_storage_remote_cache_bg_download_stage_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (stage, file_type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache BG Download Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -21924,6 +31293,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -21932,43 +31302,61 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 50 + "y": 16 }, - "hiddenSeries": false, - "id": 375, + "height": null, + "hideTimeOverride": false, + "id": 214, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, - "current": false, + "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -21981,52 +31369,131 @@ }, "lines": true, "linewidth": 1, - "nullPointMode": "null", + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, sum(rate(tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, result, file_type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, result, file_type, $additional_groupby) / 1000000000)", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "max-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, result, file_type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", - "legendFormat": "999%-{{result}}-{{file_type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "A" + "intervalFactor": 2, + "legendFormat": "80-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, result, file_type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_remote_cache_wait_on_downloading_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, result, file_type, $additional_groupby))", - "hide": false, + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) / sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, "interval": "", - "legendFormat": "99%-{{result}}-{{file_type}} {{$additional_groupby}}", - "queryType": "randomWalk", - "refId": "C" + "intervalFactor": 2, + "legendFormat": "avg-{{result}}-{{file_type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) / sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Wait on Downloading Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22034,6 +31501,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -22042,49 +31510,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 58 + "y": 24 }, - "hiddenSeries": false, - "id": 371, + "height": null, + "hideTimeOverride": false, + "id": 215, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -22093,49 +31578,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache_wait_on_downloading_result{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (result, file_type , $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_result\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{result}}-{{file_type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_result\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Wait on Downloading OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22152,6 +31648,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -22162,39 +31659,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 58 + "y": 24 }, - "hiddenSeries": false, - "id": 372, + "height": null, + "hideTimeOverride": false, + "id": 216, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -22203,44 +31716,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache_wait_on_downloading_bytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (result, file_type , $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{result}}-{{file_type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_remote_cache_wait_on_downloading_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (result, file_type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Wait on Downloading Flow", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22257,50 +31781,66 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 66 + "y": 32 }, - "hiddenSeries": false, - "id": 370, + "height": null, + "hideTimeOverride": false, + "id": 217, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -22309,44 +31849,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_storage_remote_cache_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_storage_remote_cache_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, instance) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_storage_remote_cache_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Gauge", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22354,7 +31905,7 @@ }, "yaxes": [ { - "decimals": 0, + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -22363,49 +31914,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 66 + "y": 32 }, - "hiddenSeries": false, - "id": 374, + "height": null, + "hideTimeOverride": false, + "id": 218, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -22414,49 +31982,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_remote_cache_reject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (reason, file_type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_remote_cache_reject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (reason, file_type, $additional_groupby) ", "format": "time_series", + "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{reason}}-{{file_type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_remote_cache_reject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (reason, file_type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Reject Download Type OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22473,6 +32052,7 @@ "show": true }, { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -22483,39 +32063,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Remote Cache Usage", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 74 + "y": 40 }, - "hiddenSeries": false, - "id": 188, + "height": null, + "hideTimeOverride": false, + "id": 219, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "current", "sortDesc": true, "total": false, @@ -22524,67 +32120,100 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_DTFileCacheCapacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_DTFileCacheCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "DTFileCapacity-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_DTFileCacheCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_DTFileCacheUsed{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_DTFileCacheUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "DTFileUsed-{{instance}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_DTFileCacheUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_PageCacheCapacity{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_PageCacheCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "PageCapacity-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_PageCacheCapacity\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_PageCacheUsed{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_PageCacheUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "PageUsed-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_PageCacheUsed\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Remote Cache Usage", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22601,115 +32230,168 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Memory Usage of Storage Tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 74 + "y": 40 }, - "hiddenSeries": false, - "id": 233, + "height": null, + "hideTimeOverride": false, + "id": 220, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": false, + "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_MemoryTrackingQueryStorageTask{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_MemoryTrackingQueryStorageTask\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "MemoryTrackingQueryStorageTask-{{instance}}", - "refId": "C" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_MemoryTrackingQueryStorageTask\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_MemoryTrackingFetchPages{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_MemoryTrackingFetchPages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "MemoryTrackingFetchPages-{{instance}}", - "refId": "D" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_MemoryTrackingFetchPages\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_DT_DeltaIndexCacheSize{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_DT_DeltaIndexCacheSize\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "DeltaIndexCacheSize-{{instance}}", - "refId": "E" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_DT_DeltaIndexCacheSize\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_system_current_metric_MemoryTrackingSharedColumnData{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_MemoryTrackingSharedColumnData\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "SharedColumnData-{{instance}}", - "refId": "F" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_MemoryTrackingSharedColumnData\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Memory Usage of Storage Tasks", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22726,106 +32408,143 @@ "show": true }, { - "format": "percentunit", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "DeltaIndex cache of ReadNodes", - "fieldConfig": { - "defaults": {}, - "overrides": [] + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 82 + "y": 48 }, - "hiddenSeries": false, - "id": 236, + "height": null, + "hideTimeOverride": false, + "id": 221, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, - "avg": true, + "avg": false, "current": true, + "hideEmpty": true, "hideZero": true, - "max": false, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/hit_ratio/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_mvcc_index_cache\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}-{{instance}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_mvcc_index_cache\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, instance) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"hit\"}[1m])) by (instance) /sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"hit\"}[$__rate_interval])) by (instance) / sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "hit_ratio-{{instance}}", - "refId": "A" + "metric": "", + "query": "(sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"hit\"}[$__rate_interval])) by (instance) / sum(rate(tiflash_storage_mvcc_index_cache{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "MVCCIndexCache", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22842,110 +32561,198 @@ "show": true }, { + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": true } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Duration of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } + }, + "fill": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" }, - "fill": 0, - "fillGradient": 0, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 82 + "y": 48 }, - "hiddenSeries": false, - "id": 356, + "height": null, + "hideTimeOverride": false, + "id": 222, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": null, - "sortDesc": null, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": false, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"place_index_update\"}[$__rate_interval]))) by (le,type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "max-{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"place_index_update\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "9999-{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_subtask_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"place_index_update\"}[$__rate_interval])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "99-{{type}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_subtask_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_subtask_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_subtask_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PlaceIndex Tasks Duration", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -22953,7 +32760,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -22962,101 +32769,138 @@ "show": true }, { - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Total number of storage's internal sub tasks", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 89 + "y": 56 }, - "hiddenSeries": false, - "id": 353, + "height": null, + "hideTimeOverride": false, + "id": 223, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, - "steppedLine": true, + "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_place_index_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_place_index_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_storage_place_index_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_subtask_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\"place_index_update\"}[$__rate_interval])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_storage_subtask_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\"place_index_update\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PlaceIndexTask/Reuse OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23073,49 +32917,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 89 + "y": 56 }, - "hiddenSeries": false, - "id": 358, + "height": null, + "hideTimeOverride": false, + "id": 224, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23124,64 +32985,85 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_place_index_stats_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_place_index_stats_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_place_index_stats_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_place_index_stats_count_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_place_index_stats_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "99-{{type}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_place_index_stats_count_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))\n/\nsum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "avg-{{type}} {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval])) / sum by (type, $additional_groupby) (rate(tiflash_storage_place_index_stats_count_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "PlaceIndex update rows/deletes", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23189,6 +33071,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -23197,64 +33080,108 @@ "show": true }, { - "decimals": 2, - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Disaggregated-Compute", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 19 + "y": 0 }, - "id": 172, + "height": null, + "hideTimeOverride": false, + "id": 225, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 read/write throughput", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 36 + "y": 0 }, - "hiddenSeries": false, - "id": 178, + "height": null, + "hideTimeOverride": false, + "id": 226, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23263,60 +33190,85 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3WriteBytes {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3ReadBytes {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteDMFileBytes{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteDMFileBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3WriteDMFileBytes {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteDMFileBytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 Bytes", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23333,48 +33285,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 OPS", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 36 + "y": 0 }, - "hiddenSeries": false, - "id": 179, + "height": null, + "hideTimeOverride": false, + "id": 227, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23383,132 +33353,220 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3PutObject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3PutObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3PutObject {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3PutObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3GetObject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3GetObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3GetObject {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3GetObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3HeadObject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3HeadObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3HeadObject {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3HeadObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ListObjects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ListObjects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3ListObjects {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ListObjects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3DeleteObject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3DeleteObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3DeleteObject {{$additional_groupby}}", - "refId": "E" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3DeleteObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3CopyObject{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3CopyObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3CopyObject {{$additional_groupby}}", - "refId": "F" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3CopyObject\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3CreateMultipartUpload{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3CreateMultipartUpload\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3CreateMultipartUpload {{$additional_groupby}}", - "refId": "G" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3CreateMultipartUpload\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3UploadPart{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3UploadPart\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3UploadPart {{$additional_groupby}}", - "refId": "H" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3UploadPart\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3CompleteMultipartUpload{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3CompleteMultipartUpload\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3CompleteMultipartUpload {{$additional_groupby}}", - "refId": "I" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3CompleteMultipartUpload\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3PutDMFile{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3PutDMFile\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3PutDMFile {{$additional_groupby}}", - "refId": "J" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3PutDMFile\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IORead{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IORead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3IORead {{$additional_groupby}}", - "refId": "K" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IORead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeek{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeek\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3IOSeek {{$additional_groupby}}", - "refId": "L" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeek\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23525,48 +33583,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 Retry OPS", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 44 + "y": 8 }, - "hiddenSeries": false, - "id": 182, + "height": null, + "hideTimeOverride": false, + "id": 228, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23575,90 +33651,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3GetObjectRetry{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3GetObjectRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3GetObjectRetry {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3GetObjectRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3PutObjectRetry{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3PutObjectRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3PutObjectRetry {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3PutObjectRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3PutDMFileRetry{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3PutDMFileRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3PutDMFileRetry {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3PutDMFileRetry\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOReadError{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOReadError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOReadError {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOReadError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeekError{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeekError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOSeekError {{$additional_groupby}}", - "refId": "E" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeekError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeekBackward{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeekBackward\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOSeekBackward {{$additional_groupby}}", - "refId": "F" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeekBackward\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 Retry OPS", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23675,48 +33791,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 Request Duration", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 44 + "y": 8 }, - "hiddenSeries": false, - "id": 180, + "height": null, + "hideTimeOverride": false, + "id": 229, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23725,62 +33859,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}-max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_storage_s3_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}-9999 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_s3_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}-99 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_s3_request_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_request_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_s3_request_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_request_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 Request Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23788,6 +33990,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -23796,48 +33999,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 HTTP OPS", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 52 + "y": 16 }, - "hiddenSeries": false, - "id": 343, + "height": null, + "hideTimeOverride": false, + "id": 230, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -23846,114 +34067,190 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadRequestsCount{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsCount\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "read-count {{$additional_groupby}}", - "refId": "K" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsCount\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteRequestsCount{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsCount\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "write-count {{$additional_groupby}}", - "refId": "L" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsCount\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadRequestsErrors{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsErrors\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "read-error {{$additional_groupby}}", - "refId": "M" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsErrors\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteRequestsErrors{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsErrors\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "write-error {{$additional_groupby}}", - "refId": "N" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsErrors\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadRequestsThrottling{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsThrottling\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "read-throttling {{$additional_groupby}}", - "refId": "O" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsThrottling\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteRequestsThrottling{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsThrottling\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "write-throttling {{$additional_groupby}}", - "refId": "P" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsThrottling\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadRequestsRedirects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsRedirects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "read-redirects {{$additional_groupby}}", - "refId": "Q" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsRedirects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteRequestsRedirects{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsRedirects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "write-redirects {{$additional_groupby}}", - "refId": "R" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsRedirects\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3ReadRequestsNotFound{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsNotFound\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "read-notfound {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3ReadRequestsNotFound\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3WriteRequestsNotFound{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsNotFound\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "write-notfound {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3WriteRequestsNotFound\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 HTTP OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -23970,48 +34267,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 HTTP Request Duration", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 52 + "y": 16 }, - "hiddenSeries": false, - "id": 344, + "height": null, + "hideTimeOverride": false, + "id": 231, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -24020,60 +34335,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_http_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type, $additional_groupby) / 1000000000)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_http_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}-max {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_storage_s3_http_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.9999, sum(rate(tiflash_storage_s3_http_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}-9999 {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_storage_s3_http_request_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-999 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{type}}-99 {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-80 {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_storage_s3_http_request_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, type, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_storage_s3_http_request_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_http_request_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "{{type}}-avg {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_storage_s3_http_request_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_storage_s3_http_request_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 HTTP Request Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24081,6 +34466,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -24089,48 +34475,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "S3 HTTP OPS", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 60 + "y": 24 }, - "hiddenSeries": false, - "id": 360, + "height": null, + "hideTimeOverride": false, + "id": 232, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -24139,50 +34543,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (type, $additional_groupby) (tiflash_system_current_metric_S3Requests{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_S3Requests\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3Requests {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_S3Requests\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum by (type, $additional_groupby) (tiflash_system_current_metric_S3RandomAccessFile{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_system_current_metric_S3RandomAccessFile\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "S3RandomAccessFile {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_system_current_metric_S3RandomAccessFile\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3 on-going instances", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24199,49 +34623,66 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 60 + "y": 24 }, - "hiddenSeries": false, - "id": 359, + "height": null, + "hideTimeOverride": false, + "id": 233, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -24250,84 +34691,115 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOReadError{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOReadError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOReadError {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOReadError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeekError{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeekError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOSeekError {{$additional_groupby}}", - "refId": "E" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeekError\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeekBackward{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeekBackward\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOSeekBackward {{$additional_groupby}}", - "refId": "F" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeekBackward\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IORead{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IORead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IORead {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IORead\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_system_profile_event_S3IOSeek{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_system_profile_event_S3IOSeek\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "S3IOSeek {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(rate(\n tiflash_system_profile_event_S3IOSeek\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "S3RandomAccessFile OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24344,106 +34816,165 @@ "show": true }, { - "format": "opm", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", - "show": true + "min": null, + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "S3", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 20 + "y": 0 }, - "id": 202, + "height": null, + "hideTimeOverride": false, + "id": 234, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 162 + "y": 0 }, - "hiddenSeries": false, - "id": 224, + "height": null, + "hideTimeOverride": false, + "id": 235, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_pipeline_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\".*_task_thread_pool_size\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_task_thread_pool_size\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_task_thread_pool_size\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Thread Pool Size", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24451,6 +34982,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -24459,99 +34991,138 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 162 + "y": 0 }, - "hiddenSeries": false, - "id": 226, + "height": null, + "hideTimeOverride": false, + "id": 236, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_pipeline_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\".*_tasks_count\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_tasks_count\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_tasks_count\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_pipeline_scheduler{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=~\".*_tasks_count\"}) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_tasks_count\"}\n \n)) by (type) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "sum({{type}})", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_pipeline_scheduler\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=~\".*_tasks_count\"}\n \n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Count", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24559,6 +35130,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -24567,91 +35139,123 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 170 + "y": 8 }, - "hiddenSeries": false, - "id": 228, + "height": null, + "hideTimeOverride": false, + "id": 237, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_change_to_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (type)\r\n", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_pipeline_task_change_to_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "sum(rate(\n tiflash_pipeline_task_change_to_status\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Status Change OPS", "tooltip": { + "msResolution": true, "shared": true, - "sort": 0, + "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24659,6 +35263,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -24667,52 +35272,67 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 170 + "y": 8 }, - "hiddenSeries": false, - "id": 222, + "height": null, + "hideTimeOverride": false, + "id": 238, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -24720,111 +35340,205 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", "format": "time_series", + "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "95-{{type}}", + "legendFormat": "max-{{type}} {{$additional_groupby}}", "metric": "", - "refId": "A", - "step": 4 + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "99-{{type}}", - "refId": "B", - "step": 4 + "legendFormat": "9999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", + "hide": true, + "instant": false, "interval": "", "intervalFactor": 2, - "legendFormat": "999-{{type}}", - "refId": "C", - "step": 4 + "legendFormat": "999-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[1m]))\r\n\r\n\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "99-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "80-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_pipeline_task_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_pipeline_task_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_pipeline_task_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "avg-{{type}} {{$additional_groupby}}", + "metric": "", + "query": "(sum(rate(\n tiflash_pipeline_task_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) / sum(rate(\n tiflash_pipeline_task_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[$__rate_interval])))", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "avg-cpu_execute", - "refId": "D", - "step": 4 + "metric": "", + "query": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_execute\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[1m]))\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[$__rate_interval])))", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "avg-cpu_queue", - "refId": "E", - "step": 4 + "metric": "", + "query": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu_queue\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[1m]))\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-io_execute", - "refId": "F" + "metric": "", + "query": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_execute\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[1m]))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-io_queue", - "refId": "G" + "metric": "", + "query": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io_queue\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[1m])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[1m]))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[$__rate_interval])))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-await", - "refId": "H" + "metric": "", + "query": "(sum(rate(tiflash_pipeline_task_duration_seconds_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_duration_seconds_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"await\"}[$__rate_interval])))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Duration", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 0, - "value_type": "cumulative" + "sort": 2, + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24832,6 +35546,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -24840,52 +35555,67 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 178 + "y": 16 }, - "hiddenSeries": false, - "id": 230, + "height": null, + "hideTimeOverride": false, + "id": 239, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sort": "current", + "sideWidth": null, + "sort": "max", "sortDesc": true, "total": false, "values": true @@ -24893,92 +35623,130 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "95-{{type}}", "metric": "", - "refId": "A", - "step": 4 + "query": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.95, sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "99-{{type}}", - "refId": "B", - "step": 4 + "metric": "", + "query": "histogram_quantile(0.95,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (le, type))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "999-{{type}}", - "refId": "C", - "step": 4 + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (type, le) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m]))) by (le, type) / 1000000000)\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type) / 1000000000)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "100-{{type}}", - "refId": "D" + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_bucket{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, type) / 1000000000)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[1m])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[1m]))\r\n\r\n\r\n\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[$__rate_interval]))", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "avg-cpu", - "refId": "E", - "step": 4 + "metric": "", + "query": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"cpu\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[1m])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[1m]))\r", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[$__rate_interval]))", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg-io", - "refId": "F" + "metric": "", + "query": "sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_sum{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[$__rate_interval])) / sum(rate(tiflash_pipeline_task_execute_max_time_seconds_per_round_count{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"io\"}[$__rate_interval]))", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Task Max Execute Time Per Round", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, - "value_type": "cumulative" + "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -24986,6 +35754,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -24994,53 +35763,66 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 178 + "y": 16 }, - "hiddenSeries": false, - "id": 218, + "height": null, + "hideTimeOverride": false, + "id": 240, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -25049,15 +35831,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -25068,46 +35855,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$tiflash_role\"}) by (instance)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"cpu_pool\", instance=~\"$tiflash_role\"}) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads CPU of CPU Task Thread Pool", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25115,7 +35911,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -25124,6 +35920,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -25134,43 +35931,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 186 + "y": 24 }, - "hiddenSeries": false, - "id": 220, + "height": null, + "hideTimeOverride": false, + "id": 241, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -25179,15 +35988,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -25198,46 +36012,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$tiflash_role\"}) by (instance)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"io_pool\", instance=~\"$tiflash_role\"}) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads CPU of IO Task Thread Pool", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25245,7 +36068,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -25254,6 +36077,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -25264,43 +36088,55 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", - "decimals": null, - "description": "", + "description": null, "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 186 + "y": 24 }, - "hiddenSeries": false, - "id": 216, + "height": null, + "hideTimeOverride": false, + "id": 242, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, - "sideWidth": 250, + "sideWidth": null, "sort": "max", "sortDesc": true, "total": false, @@ -25309,15 +36145,20 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "Limit", @@ -25328,46 +36169,55 @@ "nullPointMode": "connected" } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (instance) (rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[1m]))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "{{name}} {{instance}}", - "refId": "A", - "step": 40 + "metric": "", + "query": "sum(rate(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$tiflash_role\"}[$__rate_interval])) by (instance)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "count by (instance) (tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"})", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$tiflash_role\"}) by (instance)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "Limit", - "refId": "B" + "metric": "", + "query": "count(tiflash_proxy_thread_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", name=~\"WaitReactor\", instance=~\"$tiflash_role\"}) by (instance)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Threads CPU of Wait Reactor", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25375,7 +36225,7 @@ }, "yaxes": [ { - "decimals": 1, + "decimals": null, "format": "percentunit", "label": null, "logBase": 1, @@ -25384,6 +36234,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -25394,89 +36245,127 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "wait notify task details", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 194 + "y": 32 }, - "hiddenSeries": false, - "id": 300, + "height": null, + "hideTimeOverride": false, + "id": 243, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_pipeline_wait_on_notify_tasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance, type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_pipeline_wait_on_notify_tasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_pipeline_wait_on_notify_tasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (instance, type) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(tiflash_pipeline_wait_on_notify_tasks{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (type)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum((\n tiflash_pipeline_wait_on_notify_tasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "sum({{type}})", - "refId": "B" + "metric": "", + "query": "sum((\n tiflash_pipeline_wait_on_notify_tasks\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) by (type) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Wait notify task details", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25484,6 +36373,7 @@ }, "yaxes": [ { + "decimals": null, "format": "none", "label": null, "logBase": 1, @@ -25492,169 +36382,285 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Pipeline Model", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 21 + "y": 0 }, - "id": 248, + "height": null, + "hideTimeOverride": false, + "id": 244, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "description": "Metas of resource group", + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 203 + "y": 0 }, - "hiddenSeries": false, - "id": 246, + "height": null, + "hideTimeOverride": false, + "id": 245, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": true, "current": true, - "hideEmpty": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "current", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, + "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", - "pointradius": 2, + "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "max(tiflash_resource_group{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"remaining_tokens\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"remaining_tokens\"}\n \n)) by (instance, resource_group) ", + "format": "time_series", + "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "remaining_tokens-{{instance}}-{{resource_group}}", - "queryType": "randomWalk", - "refId": "A" + "metric": "", + "query": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"remaining_tokens\"}\n \n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_resource_group{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"avg_speed\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"avg_speed\"}\n \n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "avg_speed-{{instance}}-{{resource_group}}", - "refId": "B" + "metric": "", + "query": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"avg_speed\"}\n \n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_resource_group_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"total_consumption\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"total_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "total_consumption-{{instance}}-{{resource_group}}", - "refId": "C" + "metric": "", + "query": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"total_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_resource_group{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"bucket_fill_rate\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"bucket_fill_rate\"}\n \n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "bucket_fill_rate-{{instance}}-{{resource_group}}", - "refId": "D" + "metric": "", + "query": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"bucket_fill_rate\"}\n \n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "max(tiflash_resource_group{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"bucket_capacity\", instance=~\"$instance\", instance=~\"$tiflash_role\"}) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"bucket_capacity\"}\n \n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "bucket_capacity-{{instance}}-{{resource_group}}", - "refId": "E" + "metric": "", + "query": "max((\n tiflash_resource_group\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"bucket_capacity\"}\n \n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_resource_group_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"request_gac_count\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"request_gac_count\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "request_gac_count-{{instance}}-{{resource_group}}", - "refId": "F" + "metric": "", + "query": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"request_gac_count\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_resource_group_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"gac_req_ru_consumption_delta\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"gac_req_ru_consumption_delta\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "gac_req_ru_consumption_delta-{{instance}}-{{resource_group}}", - "refId": "G" + "metric": "", + "query": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"gac_req_ru_consumption_delta\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_resource_group_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"compute_ru_consumption\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"compute_ru_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "compute_ru_consumption-{{instance}}-{{resource_group}}", - "refId": "H" + "metric": "", + "query": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"compute_ru_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_resource_group_counter{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", type=\"storage_ru_consumption\", instance=~\"$instance\", instance=~\"$tiflash_role\"}[1m])) by (instance,resource_group)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"storage_ru_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "format": "time_series", "hide": true, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "storage_ru_consumption-{{instance}}-{{resource_group}}", - "refId": "I" + "metric": "", + "query": "sum(rate(\n tiflash_resource_group_counter\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"storage_ru_consumption\"}\n [$__rate_interval]\n)) by (instance, resource_group) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "TiFlash Resource Group", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25662,6 +36668,7 @@ }, "yaxes": [ { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -25670,50 +36677,62 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, "description": "Request Unit for tidb-serverless charging", "editable": true, "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, - "grid": {}, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 203 + "y": 0 }, - "hiddenSeries": false, - "id": 201, + "height": null, + "hideTimeOverride": false, + "id": 246, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": false, "min": false, "rightSide": true, @@ -25727,82 +36746,120 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": false + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/sum/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_sync_replica_ru{instance=~\"$tiflash_role\"}[1m])) by (keyspace_id, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_storage_sync_replica_ru{instance=~\"$tiflash_role\"}[$__rate_interval])) by (keyspace_id, $additional_groupby)", "format": "time_series", + "hide": false, + "instant": false, "interval": "", "intervalFactor": 2, "legendFormat": "replica-sync-rate-{{keyspace_id}}", - "refId": "A", - "step": 10 + "metric": "", + "query": "sum(rate(tiflash_storage_sync_replica_ru{instance=~\"$tiflash_role\"}[$__rate_interval])) by (keyspace_id, $additional_groupby)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "sum(increase(tiflash_storage_sync_replica_ru{instance=~\"$tiflash_role\"}[24h])) by (keyspace_id, $additional_groupby)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "replica-sync-sum-24h-{{keyspace_id}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "sum(increase(tiflash_storage_sync_replica_ru{instance=~\"$tiflash_role\"}[24h])) by (keyspace_id, $additional_groupby)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_compute_request_unit{instance=~\"$tiflash_role\"}[1m])) by (cluster_id, $additional_groupby)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_compute_request_unit{instance=~\"$tiflash_role\"}[$__rate_interval])) by (cluster_id, $additional_groupby)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "query-rate-{{cluster_id}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "sum(rate(tiflash_compute_request_unit{instance=~\"$tiflash_role\"}[$__rate_interval])) by (cluster_id, $additional_groupby)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, + "datasource": "${DS_TEST-CLUSTER}", "expr": "sum(increase(tiflash_compute_request_unit{instance=~\"$tiflash_role\"}[24h])) by (cluster_id, $additional_groupby)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "query-sum-24h-{{cluster_id}} {{$additional_groupby}}", - "refId": "D" + "metric": "", + "query": "sum(increase(tiflash_compute_request_unit{instance=~\"$tiflash_role\"}[24h])) by (cluster_id, $additional_groupby)", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "sum(rate(tiflash_storage_ru_read_bytes{instance=~\"$tiflash_role\"}[1m])) by (keyspace, resource_group, type, $additional_groupby) / (64 * 1024)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate(tiflash_storage_ru_read_bytes{instance=~\"$tiflash_role\"}[$__rate_interval])) by (keyspace, resource_group, type, $additional_groupby) / (64 * 1024)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "storage-{{keyspace}}_{{resource_group}}_{{type}} {{$additional_groupby}}", - "refId": "E" + "metric": "", + "query": "sum(rate(tiflash_storage_ru_read_bytes{instance=~\"$tiflash_role\"}[$__rate_interval])) by (keyspace, resource_group, type, $additional_groupby) / (64 * 1024)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Request Unit", "tooltip": { - "msResolution": false, + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25810,6 +36867,7 @@ }, "yaxes": [ { + "decimals": null, "format": "cps", "label": null, "logBase": 1, @@ -25818,6 +36876,7 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, @@ -25828,116 +36887,229 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "TiFlash Resource Control", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 22 + "y": 0 }, - "id": 340, + "height": null, + "hideTimeOverride": false, + "id": 247, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, - "bars": false, - "dashLength": 10, - "dashes": false, + "bars": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 0, - "y": 204 + "y": 0 }, - "hiddenSeries": false, - "id": 338, + "height": null, + "hideTimeOverride": false, + "id": 248, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": true, - "max": false, + "hideEmpty": true, + "hideZero": true, + "max": true, "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], - "nullPointMode": "null", + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, + "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(0.999, \n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[$__rate_interval]\n)) by (le, path, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, path, $additional_groupby) / 1000000000)", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, + "legendFormat": "max-{{path}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(1.00, sum(round(1000000000*rate(tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket{k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}[$__rate_interval]))) by (le, path, $additional_groupby) / 1000000000)", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": false, + "instant": false, + "interval": "", + "intervalFactor": 2, + "legendFormat": "9999-{{path}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.9999,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "999-{{path}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile(0.999,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "histogram_quantile(0.99, \n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}[$__rate_interval]\n)) by (le, path, $additional_groupby))", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, "legendFormat": "99-{{path}} {{$additional_groupby}}", - "refId": "B" + "metric": "", + "query": "histogram_quantile(0.99,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "(sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) / sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) )", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", "format": "time_series", "hide": true, + "instant": false, "interval": "", - "intervalFactor": 1, + "intervalFactor": 2, + "legendFormat": "80-{{path}} {{$additional_groupby}}", + "metric": "", + "query": "histogram_quantile(0.80,(\n sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_bucket\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (le, path, $additional_groupby) \n \n \n)) ", + "refId": "", + "step": 10, + "target": "" + }, + { + "datasource": "${DS_TEST-CLUSTER}", + "expr": "(sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) / sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) )", + "format": "time_series", + "hide": true, + "instant": false, + "interval": "", + "intervalFactor": 2, "legendFormat": "avg-{{path}} {{$additional_groupby}}", - "refId": "C" + "metric": "", + "query": "(sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_sum\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) / sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$proxy_instance\",instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Status API Request Duration", "tooltip": { + "msResolution": true, "shared": true, "sort": 2, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -25945,6 +37117,7 @@ }, "yaxes": [ { + "decimals": null, "format": "s", "label": null, "logBase": 1, @@ -25953,39 +37126,55 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 1, - "fillGradient": 1, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { - "h": 7, + "h": 8, "w": 12, "x": 12, - "y": 204 + "y": 0 }, - "hiddenSeries": false, - "id": 341, + "height": null, + "hideTimeOverride": false, + "id": 249, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, @@ -25996,48 +37185,64 @@ "min": false, "rightSide": true, "show": true, + "sideWidth": null, + "sort": "max", + "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum(rate(\n tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"}\n [$__rate_interval]\n)) by (path, $additional_groupby) ", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum(rate( tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"} [$__rate_interval] )) by (path, $additional_groupby)", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{path}} {{$additional_groupby}}", - "refId": "A" + "metric": "", + "query": "sum(rate( tiflash_proxy_tikv_status_server_proxy_request_duration_seconds_count {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\", instance=~\"$proxy_instance\", instance=~\"$tiflash_role\"} [$__rate_interval] )) by (path, $additional_groupby)", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Status API Request (op/s)", "tooltip": { + "msResolution": true, "shared": true, - "sort": 2, + "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -26045,6 +37250,7 @@ }, "yaxes": [ { + "decimals": null, "format": "ops", "label": null, "logBase": 1, @@ -26053,69 +37259,98 @@ "show": true }, { + "decimals": null, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, - "show": true + "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Status Server", + "transformations": [], + "transparent": false, "type": "row" }, { + "cacheTimeout": null, "collapsed": true, "datasource": null, + "description": null, + "editable": true, + "error": false, + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + } + } + }, "gridPos": { - "h": 1, + "h": 8, "w": 24, "x": 0, - "y": 23 + "y": 0 }, - "id": 274, + "height": null, + "hideTimeOverride": false, + "id": 250, + "interval": null, + "links": [], + "maxDataPoints": 100, + "maxPerRow": null, + "minSpan": null, "panels": [ { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 0, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "/delete_range|ingest/" - }, - "properties": [ - { - "id": "unit", - "value": "opm" - } - ] - } - ] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 205 + "y": 0 }, - "hiddenSeries": false, - "id": 286, + "height": null, + "hideTimeOverride": false, + "id": 251, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, @@ -26135,44 +37370,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "sum by (type, instance) (\n tiflash_vector_index_active_instances{\n k8s_cluster=\"$k8s_cluster\", \n tidb_cluster=\"$tidb_cluster\", \n instance=~\"$instance\", instance=~\"$tiflash_role\"\n }\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "sum by (type, instance) ( tiflash_vector_index_active_instances{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\" } )", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "sum by (type, instance) ( tiflash_vector_index_active_instances{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\" } )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "In-Memory Vector Index Instances", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -26182,61 +37428,63 @@ { "decimals": 0, "format": "short", - "label": "", + "label": null, "logBase": 1, "max": null, "min": "0", "show": true }, { - "format": "ops", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 0, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "/delete_range|ingest/" - }, - "properties": [ - { - "id": "unit", - "value": "opm" - } - ] - } - ] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 205 + "y": 0 }, - "hiddenSeries": false, - "id": 288, + "height": null, + "hideTimeOverride": false, + "id": 252, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, @@ -26256,52 +37504,70 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "tiflash_vector_index_memory_usage{\n k8s_cluster=\"$k8s_cluster\", \n tidb_cluster=\"$tidb_cluster\", \n instance=~\"$instance\", instance=~\"$tiflash_role\"\n}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_vector_index_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{instance}}-{{type}}", - "refId": "A" + "metric": "", + "query": "((\n tiflash_vector_index_memory_usage\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" }, { - "exemplar": true, - "expr": "tiflash_process_rss_by_type_bytes{\n k8s_cluster=\"$k8s_cluster\", \n tidb_cluster=\"$tidb_cluster\", \n instance=~\"$instance\", instance=~\"$tiflash_role\",\n type=\"file\"\n}", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "((\n tiflash_process_rss_by_type_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"file\"}\n \n)) ", + "format": "time_series", "hide": false, + "instant": false, "interval": "", + "intervalFactor": 2, "legendFormat": "{{instance}}-RssFile", - "refId": "B" + "metric": "", + "query": "((\n tiflash_process_rss_by_type_bytes\n {k8s_cluster=\"$k8s_cluster\",tidb_cluster=\"$tidb_cluster\",instance=~\"$instance\",instance=~\"$tiflash_role\",type=\"file\"}\n \n)) ", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "Vector Index Estimated Memory Usage", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -26318,60 +37584,62 @@ "show": true }, { - "format": "ops", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "/delete_range|ingest/" - }, - "properties": [ - { - "id": "unit", - "value": "opm" - } - ] - } - ] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 0, - "y": 213 + "y": 8 }, - "hiddenSeries": false, - "id": 282, + "height": null, + "hideTimeOverride": false, + "id": 253, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": false, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -26385,49 +37653,60 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [ { "alias": "/download/", "yaxis": 2 } ], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(\n 0.999, \n sum(rate(\n tiflash_vector_index_duration_bucket{\n k8s_cluster=\"$k8s_cluster\", \n tidb_cluster=\"$tidb_cluster\", \n instance=~\"$instance\", instance=~\"$tiflash_role\", \n type!=\"build\"\n }\n [$__rate_interval]\n )) by (le, type)\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!=\"build\" } [$__rate_interval] )) by (le, type) )", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type!=\"build\" } [$__rate_interval] )) by (le, type) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "99.9% Vector Search Duration (Per Request)", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -26455,50 +37734,51 @@ ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } }, { "aliasColors": {}, "bars": false, - "dashLength": 10, - "dashes": false, + "cacheTimeout": null, "datasource": "${DS_TEST-CLUSTER}", "decimals": 1, - "description": "", + "description": null, + "editable": true, + "error": false, "fieldConfig": { - "defaults": {}, - "overrides": [ - { - "matcher": { - "id": "byName", - "options": "/delete_range|ingest/" - }, - "properties": [ - { - "id": "unit", - "value": "opm" - } - ] - } - ] + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [] + }, + "unit": "" + } }, "fill": 0, - "fillGradient": 0, + "grid": { + "threshold1": null, + "threshold1Color": "rgba(216, 200, 27, 0.27)", + "threshold2": null, + "threshold2Color": "rgba(234, 112, 112, 0.22)" + }, "gridPos": { "h": 8, "w": 12, "x": 12, - "y": 213 + "y": 8 }, - "hiddenSeries": false, - "id": 284, + "height": null, + "hideTimeOverride": false, + "id": 254, + "interval": null, + "isNew": true, "legend": { "alignAsTable": true, "avg": false, "current": false, - "hideEmpty": false, - "hideZero": false, + "hideEmpty": true, + "hideZero": true, "max": true, "min": false, "rightSide": true, @@ -26512,44 +37792,55 @@ "lines": true, "linewidth": 1, "links": [], + "maxDataPoints": null, + "maxPerRow": null, + "minSpan": null, "nullPointMode": "null as zero", "options": { - "alertThreshold": true + "alertThreshold": true, + "dataLinks": [] }, "percentage": false, - "pluginVersion": "7.5.11", "pointradius": 5, "points": false, "renderer": "flot", + "repeat": null, + "repeatDirection": null, "seriesOverrides": [], - "spaceLength": 10, + "span": null, "stack": false, "steppedLine": false, "targets": [ { - "exemplar": true, - "expr": "histogram_quantile(\n 0.999, \n sum(rate(\n tiflash_vector_index_duration_bucket{\n k8s_cluster=\"$k8s_cluster\", \n tidb_cluster=\"$tidb_cluster\", \n instance=~\"$instance\", instance=~\"$tiflash_role\", \n type=\"build\"\n }\n [$__rate_interval]\n )) by (le, type)\n)", + "datasource": "${DS_TEST-CLUSTER}", + "expr": "histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"build\" } [$__rate_interval] )) by (le, type) )", "format": "time_series", "hide": false, + "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{type}}", - "refId": "A" + "metric": "", + "query": "histogram_quantile( 0.999, sum(rate( tiflash_vector_index_duration_bucket{ k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", instance=~\"$instance\", instance=~\"$tiflash_role\", type=\"build\" } [$__rate_interval] )) by (le, type) )", + "refId": "", + "step": 10, + "target": "" } ], "thresholds": [], "timeFrom": null, - "timeRegions": [], "timeShift": null, "title": "99.9% Vector Index Build Duration (Per DMFile Column)", "tooltip": { + "msResolution": true, "shared": true, "sort": 0, "value_type": "individual" }, + "transformations": [], + "transparent": false, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", "name": null, "show": true, @@ -26566,148 +37857,160 @@ "show": true }, { - "decimals": 1, - "format": "s", + "decimals": null, + "format": "short", "label": null, "logBase": 1, "max": null, - "min": "0", + "min": null, "show": false } ], "yaxis": { "align": false, - "alignLevel": null + "alignLevel": 0 } } ], + "repeat": null, + "repeatDirection": null, + "span": null, + "targets": [], + "timeFrom": null, + "timeShift": null, "title": "Vector Search", + "transformations": [], + "transparent": false, "type": "row" } ], "refresh": "1m", - "schemaVersion": 27, + "rows": [], + "schemaVersion": 14, + "sharedCrosshair": false, "style": "dark", "tags": [], "templating": { "list": [ { "allValue": null, - "current": {}, + "auto": false, + "auto_count": 30, + "auto_min": "10s", + "current": { + "selected": false, + "tags": [], + "text": null, + "value": null + }, "datasource": "${DS_TEST-CLUSTER}", - "definition": "", - "description": null, - "error": null, "hide": 2, "includeAll": false, "label": "K8s-cluster", "multi": false, "name": "k8s_cluster", "options": [], - "query": { - "query": "label_values(tiflash_system_profile_event_Query, k8s_cluster)", - "refId": "j2-k8s_cluster-Variable-Query" - }, + "query": "label_values(tiflash_system_profile_event_Query, k8s_cluster)", "refresh": 2, - "regex": "", - "skipUrlSync": false, + "regex": null, "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", + "tagValuesQuery": null, + "tagsQuery": null, "type": "query", "useTags": false }, { "allValue": null, - "current": {}, + "auto": false, + "auto_count": 30, + "auto_min": "10s", + "current": { + "selected": false, + "tags": [], + "text": null, + "value": null + }, "datasource": "${DS_TEST-CLUSTER}", - "definition": "", - "description": null, - "error": null, "hide": 2, "includeAll": false, "label": "tidb_cluster", "multi": false, "name": "tidb_cluster", "options": [], - "query": { - "query": "label_values(tiflash_system_profile_event_Query{k8s_cluster=\"$k8s_cluster\"}, tidb_cluster)", - "refId": "j2-tidb_cluster-Variable-Query" - }, + "query": "label_values(tiflash_system_profile_event_Query{k8s_cluster=\"$k8s_cluster\"}, tidb_cluster)", "refresh": 2, - "regex": "", - "skipUrlSync": false, + "regex": null, "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", + "tagValuesQuery": null, + "tagsQuery": null, "type": "query", "useTags": false }, { "allValue": null, - "current": {}, + "auto": false, + "auto_count": 30, + "auto_min": "10s", + "current": { + "selected": false, + "tags": [], + "text": null, + "value": null + }, "datasource": "${DS_TEST-CLUSTER}", - "definition": "label_values(tiflash_system_profile_event_Query{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", - "description": null, - "error": null, "hide": 0, "includeAll": true, "label": "Instance", "multi": true, "name": "instance", "options": [], - "query": { - "query": "label_values(tiflash_system_profile_event_Query{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", - "refId": "j2-instance-Variable-Query" - }, + "query": "label_values(tiflash_system_profile_event_Query{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", "refresh": 1, - "regex": "", - "skipUrlSync": false, + "regex": null, "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", + "tagValuesQuery": null, + "tagsQuery": null, "type": "query", "useTags": false }, { "allValue": null, - "current": {}, + "auto": false, + "auto_count": 30, + "auto_min": "10s", + "current": { + "selected": false, + "tags": [], + "text": null, + "value": null + }, "datasource": "${DS_TEST-CLUSTER}", - "definition": "label_values(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", - "description": null, - "error": null, "hide": 0, "includeAll": true, "label": "Proxy Instance", "multi": true, "name": "proxy_instance", "options": [], - "query": { - "query": "label_values(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", - "refId": "StandardVariableQuery" - }, + "query": "label_values(tiflash_proxy_process_cpu_seconds_total{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}, instance)", "refresh": 1, - "regex": "", - "skipUrlSync": false, + "regex": null, "sort": 1, - "tagValuesQuery": "", - "tags": [], - "tagsQuery": "", + "tagValuesQuery": null, + "tagsQuery": null, "type": "query", "useTags": false }, { "allValue": null, + "auto": false, + "auto_count": 30, + "auto_min": "10s", "current": { - "selected": false, + "selected": true, "text": "none", "value": "none" }, - "description": null, - "error": null, + "datasource": null, "hide": 0, "includeAll": false, "label": "additional_groupby", @@ -26726,19 +38029,25 @@ } ], "query": "none,instance", - "queryValue": "", - "skipUrlSync": false, - "type": "custom" + "refresh": 1, + "regex": null, + "sort": 1, + "tagValuesQuery": null, + "tagsQuery": null, + "type": "custom", + "useTags": false }, { "allValue": null, + "auto": false, + "auto_count": 30, + "auto_min": "10s", "current": { - "selected": false, + "selected": true, "text": "All", "value": ".*" }, - "description": null, - "error": null, + "datasource": null, "hide": 0, "includeAll": false, "label": "Role", @@ -26762,9 +38071,13 @@ } ], "query": "All : .*, Write : .*write-tiflash.*, Compute : .*compute-tiflash.*", - "queryValue": "", - "skipUrlSync": false, - "type": "custom" + "refresh": 1, + "regex": null, + "sort": 1, + "tagValuesQuery": null, + "tagsQuery": null, + "type": "custom", + "useTags": false } ] }, @@ -26773,6 +38086,8 @@ "to": "now" }, "timepicker": { + "hidden": false, + "nowDelay": null, "refresh_intervals": [ "5s", "10s", @@ -26797,8 +38112,8 @@ "30d" ] }, - "timezone": "", + "timezone": "browser", "title": "Test-Cluster-TiFlash-Summary", "uid": "SVbh2xUWk", - "version": 1 + "version": 0 } diff --git a/metrics/grafana/tiflash_summary.json.sha256 b/metrics/grafana/tiflash_summary.json.sha256 new file mode 100644 index 00000000000..7b01ad543e1 --- /dev/null +++ b/metrics/grafana/tiflash_summary.json.sha256 @@ -0,0 +1 @@ +6ed50a4a0e9d4dc79211ec7fa1a4a23e412ae04a82ba52c3a6b073b847afdd7d ./metrics/grafana/tiflash_summary.json diff --git a/metrics/grafana/uv.lock b/metrics/grafana/uv.lock new file mode 100644 index 00000000000..1f77c470f5b --- /dev/null +++ b/metrics/grafana/uv.lock @@ -0,0 +1,255 @@ +version = 1 +revision = 3 +requires-python = ">=3.10" + +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + +[[package]] +name = "black" +version = "26.5.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "mypy-extensions" }, + { name = "packaging" }, + { name = "pathspec" }, + { name = "platformdirs" }, + { name = "pytokens" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/37/5628dd55bf2b34257fc7603f0fe97c40e3aaf24265f416a9c85c95ca1436/black-26.5.1.tar.gz", hash = "sha256:dd321f668053961824bcc1be1cc1df748b2d7e4fa28086b08331e577b0100a73", size = 679439, upload-time = "2026-05-18T16:53:36.107Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/be/84/b3f55026206a9e8820a91503308075ca48eadc515e436731ca01dbe043b3/black-26.5.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9942db8888e06943c5dde66ca0037dcff82a2a4ec1ad0ada9e0d2ee9d9823893", size = 1987719, upload-time = "2026-05-18T17:05:02.757Z" }, + { url = "https://files.pythonhosted.org/packages/c6/34/7db312c5e5783d6e76cffd9d5ac8972a32badae4c6e3288dac0eed8d3bed/black-26.5.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:89c93167a74d3a75dfaa38a5c7cca015537d5820dd7f17d63267d674a61cae90", size = 1810083, upload-time = "2026-05-18T17:05:04.302Z" }, + { url = "https://files.pythonhosted.org/packages/33/e2/e0101e73c2c8727634e2efcb35e2b34bd23ad70dfa673789f5773a591b21/black-26.5.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:22f2cd76d069cc54c71f10360744ba8983fbb616903b4304a85b734915c8e1b4", size = 1860633, upload-time = "2026-05-18T17:05:06.391Z" }, + { url = "https://files.pythonhosted.org/packages/b0/4c/e15c0c5b23cf3651035fe5addcce90e283af3548a3f91bb03d81b83106ab/black-26.5.1-cp310-cp310-win_amd64.whl", hash = "sha256:87ed5c6f450580a2f6790bc7cbfb016dfc73bc750249762268a3695361315eef", size = 1477886, upload-time = "2026-05-18T17:05:07.96Z" }, + { url = "https://files.pythonhosted.org/packages/9f/3f/59d43ade98d2ce5c8dc34a4e46cbecd177e6d55d7d4092969c6003ccc655/black-26.5.1-cp310-cp310-win_arm64.whl", hash = "sha256:58b4bd92cf88aacf83d88479c8f9caee044b1ec55f2451a337354a7ea2590a22", size = 1277111, upload-time = "2026-05-18T17:05:09.473Z" }, + { url = "https://files.pythonhosted.org/packages/4b/96/3c3e09f09f44a37aac36b178a279cd19aa7001bd796187a7b162a294c81f/black-26.5.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:96ae2c733b2aabdd9986e2c5df628ff3473676cd1c5faded1ff496cf6d74083c", size = 1970639, upload-time = "2026-05-18T17:05:11.461Z" }, + { url = "https://files.pythonhosted.org/packages/83/ea/5ad117b9ee3ecd933c712bcbae610006e5b7cc9f41c526cd7ed3b6c4124c/black-26.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0e48b87e03bf109288e55cfceadcfa15ff5470aca2851a851950ed2926f450d7", size = 1792130, upload-time = "2026-05-18T17:05:12.983Z" }, + { url = "https://files.pythonhosted.org/packages/06/3a/7c448bc623fcdfa96672531beb5a616ea5e64f6975955254d7731ffb0ad9/black-26.5.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5119fa92ae61f786e8c3662fd60aece1d0a2dd5cca5d0c79417a95e7a4272a59", size = 1846134, upload-time = "2026-05-18T17:05:14.506Z" }, + { url = "https://files.pythonhosted.org/packages/a1/5b/0b39b3a5917f0657ac014ad2edb58c139553a478adfe7f817abf1622ff6e/black-26.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:30d3c14661f2792e9142cce3eeeb1cbc175b3eb5f733be0c8eeb99651e52b0c3", size = 1478883, upload-time = "2026-05-18T17:05:16.542Z" }, + { url = "https://files.pythonhosted.org/packages/4c/48/dc222692e0f95030db1bbfb6c857e76858bad09058221ea7aae815255327/black-26.5.1-cp311-cp311-win_arm64.whl", hash = "sha256:1ef92b76f7733f282fd096ea406200b5a286c42947412b0eaff3a74e3616cefe", size = 1277776, upload-time = "2026-05-18T17:05:18.029Z" }, + { url = "https://files.pythonhosted.org/packages/24/99/7744b906703228264ef73bdd534df88ec1ef3de45c4e78f6d31b9e32d0c9/black-26.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:4ad6fa01f941920f54f2bbb35f3df7673428a0ef98a0b0840c2eaef3b110efa8", size = 2012518, upload-time = "2026-05-18T17:05:20.108Z" }, + { url = "https://files.pythonhosted.org/packages/b7/c0/c5a3b1636dfd09c42534f2b3cf33506814f6d3e066fb0879ffa16c1ae860/black-26.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3915f256e75a2d7cf88d8953d37f780455dc586cc72dee059c528fe77f581217", size = 1816016, upload-time = "2026-05-18T17:05:21.84Z" }, + { url = "https://files.pythonhosted.org/packages/1f/0e/36044316b65ca471d3bb6d3703fd06fb50c6b727c3562f6a5a3153634f88/black-26.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d98d4137277c75dfb898ec8d846c4fd68ba1e9cf77f95e2865c203dc18f4c3d", size = 1884150, upload-time = "2026-05-18T17:05:23.546Z" }, + { url = "https://files.pythonhosted.org/packages/b3/33/dafc5808c2af43672912111d7c3354af1615f7e2be3bed7a878461abbe4d/black-26.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:a1dca32d9f1784af512a13410ec204c6f7f0aa9797a111c42e1c03449821c264", size = 1486825, upload-time = "2026-05-18T17:05:25.004Z" }, + { url = "https://files.pythonhosted.org/packages/82/14/b965ee6ad2a311f28bdbf692def3ee9848d2ae289dab28b27657fcee3e78/black-26.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:1037d5ac7b7b310b2632ad867ec8d0e4c4819dcdb0b820f63135da746a24e418", size = 1288646, upload-time = "2026-05-18T17:05:26.477Z" }, + { url = "https://files.pythonhosted.org/packages/3f/5c/c384363980e11e25ca6b93205949bb331fbf35f4e0dbec376dfa6326cec8/black-26.5.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2b36cf2ddf5566e205f6535f782a62194a184d33e175b64ae8c40b1737522be3", size = 2009020, upload-time = "2026-05-18T17:05:28.132Z" }, + { url = "https://files.pythonhosted.org/packages/0b/df/9f31c5e0babbfed77d505fc5d120beb98b21b33feaeded3924ea941fe360/black-26.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1f7ea64ebfa01b50f693508fc39f875e264446d3b097088f84f203b9d09618a0", size = 1813335, upload-time = "2026-05-18T17:05:31.266Z" }, + { url = "https://files.pythonhosted.org/packages/fb/24/8e7b9a2fa61b0afd82209efe937557d180a1fa055bd7f6161eb9defc3719/black-26.5.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ecb3e624844c798144e9bd986954e0adc81d8911a1f30f375e1252fe26e8c294", size = 1881614, upload-time = "2026-05-18T17:05:32.718Z" }, + { url = "https://files.pythonhosted.org/packages/49/ad/b4e0d9365ba8ac34f6bbab62a4b1b2dd5d618fac3fa1b8db968c844201b5/black-26.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:e1a26503279b6b310669fb0b219c39e4820b77e8189fe80f522bb511f247db0a", size = 1488925, upload-time = "2026-05-18T17:05:34.259Z" }, + { url = "https://files.pythonhosted.org/packages/a1/4b/652b859bf5df88a751c30451b09338f7fd26a77d1271c666992f836b7711/black-26.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:5c34b25da232ead53a6f335b76dbea124f4d152ad568b9080d6f944bc2b34b52", size = 1289883, upload-time = "2026-05-18T17:05:36.019Z" }, + { url = "https://files.pythonhosted.org/packages/a6/16/a8da8eb208c51c7f4ce74609a45d0dcc6d8a2141e45e81ee5289d1bb0d59/black-26.5.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e88976690a64b0af98312ca958415849cb42423423c5f2ee74af4b49a97a2168", size = 2004800, upload-time = "2026-05-18T17:05:38.182Z" }, + { url = "https://files.pythonhosted.org/packages/11/8a/a479296a19e383b70a725882a6cf3d786540601ff03cabbaaf1cce864c5a/black-26.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:32d5ea7f6c8bdfa6e648326ebca1f02b0764e2a029edc6f8dce2627e19d468c3", size = 1815576, upload-time = "2026-05-18T17:05:40.309Z" }, + { url = "https://files.pythonhosted.org/packages/81/6b/cfaf3d39f25132c156a068f6b805576c9103a84086019507c70e1911ee7d/black-26.5.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ea8d16dc41655aa113cd64665e7219446cd7e4ff2248d7178eaa905190c86b18", size = 1877927, upload-time = "2026-05-18T17:05:42.463Z" }, + { url = "https://files.pythonhosted.org/packages/66/76/302e313964bcff7e28df329d39f84f5270095730d85ff0acc260610a0d82/black-26.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:577f21094ea469ef92ec1adaf2c9441a226d2144d01a5be2fa823cecf6543e50", size = 1511860, upload-time = "2026-05-18T17:05:43.943Z" }, + { url = "https://files.pythonhosted.org/packages/27/4e/a3827e35e0e567f9f9ee59e2a0ab979267dca98718f25547ca8c6733afd4/black-26.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:ed1a20af114c301a0269bf01163d51dbef72737fd65f850001e7cbe7f3c7abae", size = 1316632, upload-time = "2026-05-18T17:05:45.521Z" }, + { url = "https://files.pythonhosted.org/packages/94/51/f975cae76d44274cc2868dc9040ac5d58d464784610234455b4e7b19c6ef/black-26.5.1-py3-none-any.whl", hash = "sha256:4ed7f7da04046d2e488437170797d3b4a4ad83906683bcb7dfc68b673bbce5e2", size = 213693, upload-time = "2026-05-18T16:53:33.964Z" }, +] + +[[package]] +name = "click" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "grafanalib" +version = "0.7.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b7/a4/e413c2aa91e3ea631b0493f0c1473d6a48afa1fdc2b576ae6364a372bf30/grafanalib-0.7.1.tar.gz", hash = "sha256:3d92bb4e92ae78fe4e21c5b252ab51f4fdcacd8523ba5a44545b897b2a375b83", size = 59616, upload-time = "2024-01-12T10:35:11.018Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d3/6e/825081b0ae89031ec16de0c57bf6dbb06d26b69dc938c3461c831e409c9b/grafanalib-0.7.1-py3-none-any.whl", hash = "sha256:6fab5d7b837a1f2d1322ef762cd52e565ec0422707a7512765c59f668bdceb58", size = 62780, upload-time = "2024-01-12T10:35:08.788Z" }, +] + +[[package]] +name = "isort" +version = "8.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ef/7c/ec4ab396d31b3b395e2e999c8f46dec78c5e29209fac49d1f4dace04041d/isort-8.0.1.tar.gz", hash = "sha256:171ac4ff559cdc060bcfff550bc8404a486fee0caab245679c2abe7cb253c78d", size = 769592, upload-time = "2026-02-28T10:08:20.685Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/95/c7c34aa53c16353c56d0b802fba48d5f5caa2cdee7958acbcb795c830416/isort-8.0.1-py3-none-any.whl", hash = "sha256:28b89bc70f751b559aeca209e6120393d43fbe2490de0559662be7a9787e3d75", size = 89733, upload-time = "2026-02-28T10:08:19.466Z" }, +] + +[[package]] +name = "mypy-extensions" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, +] + +[[package]] +name = "packaging" +version = "26.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412, upload-time = "2026-08-04T18:15:28.737Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956, upload-time = "2026-08-04T18:15:27.159Z" }, +] + +[[package]] +name = "pathspec" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.11.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/36/0a/062135c9a98dac804265073cc3afdbec5ae1aa37980bb354f461bafe81b4/platformdirs-4.11.1.tar.gz", hash = "sha256:bb1af68078f25e2f3e111e2d43b8d536df41b73c8a684b40bb018223b66fae27", size = 32396, upload-time = "2026-08-07T23:06:48.516Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/85/9b31b44296cfa3bb56cddb35e6a0f6578bab0b490c0806c0245e32c6110c/platformdirs-4.11.1-py3-none-any.whl", hash = "sha256:2efd27d363e8dd2e661639ffb398865a5e0a46442a11d266bf375a0e0c10e386", size = 23261, upload-time = "2026-08-07T23:06:47.219Z" }, +] + +[[package]] +name = "pytokens" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b6/34/b4e015b99031667a7b960f888889c5bd34ef585c85e1cb56a594b92836ac/pytokens-0.4.1.tar.gz", hash = "sha256:292052fe80923aae2260c073f822ceba21f3872ced9a68bb7953b348e561179a", size = 23015, upload-time = "2026-01-30T01:03:45.924Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/42/24/f206113e05cb8ef51b3850e7ef88f20da6f4bf932190ceb48bd3da103e10/pytokens-0.4.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2a44ed93ea23415c54f3face3b65ef2b844d96aeb3455b8a69b3df6beab6acc5", size = 161522, upload-time = "2026-01-30T01:02:50.393Z" }, + { url = "https://files.pythonhosted.org/packages/d4/e9/06a6bf1b90c2ed81a9c7d2544232fe5d2891d1cd480e8a1809ca354a8eb2/pytokens-0.4.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:add8bf86b71a5d9fb5b89f023a80b791e04fba57960aa790cc6125f7f1d39dfe", size = 246945, upload-time = "2026-01-30T01:02:52.399Z" }, + { url = "https://files.pythonhosted.org/packages/69/66/f6fb1007a4c3d8b682d5d65b7c1fb33257587a5f782647091e3408abe0b8/pytokens-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:670d286910b531c7b7e3c0b453fd8156f250adb140146d234a82219459b9640c", size = 259525, upload-time = "2026-01-30T01:02:53.737Z" }, + { url = "https://files.pythonhosted.org/packages/04/92/086f89b4d622a18418bac74ab5db7f68cf0c21cf7cc92de6c7b919d76c88/pytokens-0.4.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:4e691d7f5186bd2842c14813f79f8884bb03f5995f0575272009982c5ac6c0f7", size = 262693, upload-time = "2026-01-30T01:02:54.871Z" }, + { url = "https://files.pythonhosted.org/packages/b4/7b/8b31c347cf94a3f900bdde750b2e9131575a61fdb620d3d3c75832262137/pytokens-0.4.1-cp310-cp310-win_amd64.whl", hash = "sha256:27b83ad28825978742beef057bfe406ad6ed524b2d28c252c5de7b4a6dd48fa2", size = 103567, upload-time = "2026-01-30T01:02:56.414Z" }, + { url = "https://files.pythonhosted.org/packages/3d/92/790ebe03f07b57e53b10884c329b9a1a308648fc083a6d4a39a10a28c8fc/pytokens-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d70e77c55ae8380c91c0c18dea05951482e263982911fc7410b1ffd1dadd3440", size = 160864, upload-time = "2026-01-30T01:02:57.882Z" }, + { url = "https://files.pythonhosted.org/packages/13/25/a4f555281d975bfdd1eba731450e2fe3a95870274da73fb12c40aeae7625/pytokens-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a58d057208cb9075c144950d789511220b07636dd2e4708d5645d24de666bdc", size = 248565, upload-time = "2026-01-30T01:02:59.912Z" }, + { url = "https://files.pythonhosted.org/packages/17/50/bc0394b4ad5b1601be22fa43652173d47e4c9efbf0044c62e9a59b747c56/pytokens-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b49750419d300e2b5a3813cf229d4e5a4c728dae470bcc89867a9ad6f25a722d", size = 260824, upload-time = "2026-01-30T01:03:01.471Z" }, + { url = "https://files.pythonhosted.org/packages/4e/54/3e04f9d92a4be4fc6c80016bc396b923d2a6933ae94b5f557c939c460ee0/pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d9907d61f15bf7261d7e775bd5d7ee4d2930e04424bab1972591918497623a16", size = 264075, upload-time = "2026-01-30T01:03:04.143Z" }, + { url = "https://files.pythonhosted.org/packages/d1/1b/44b0326cb5470a4375f37988aea5d61b5cc52407143303015ebee94abfd6/pytokens-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:ee44d0f85b803321710f9239f335aafe16553b39106384cef8e6de40cb4ef2f6", size = 103323, upload-time = "2026-01-30T01:03:05.412Z" }, + { url = "https://files.pythonhosted.org/packages/41/5d/e44573011401fb82e9d51e97f1290ceb377800fb4eed650b96f4753b499c/pytokens-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:140709331e846b728475786df8aeb27d24f48cbcf7bcd449f8de75cae7a45083", size = 160663, upload-time = "2026-01-30T01:03:06.473Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e6/5bbc3019f8e6f21d09c41f8b8654536117e5e211a85d89212d59cbdab381/pytokens-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d6c4268598f762bc8e91f5dbf2ab2f61f7b95bdc07953b602db879b3c8c18e1", size = 255626, upload-time = "2026-01-30T01:03:08.177Z" }, + { url = "https://files.pythonhosted.org/packages/bf/3c/2d5297d82286f6f3d92770289fd439956b201c0a4fc7e72efb9b2293758e/pytokens-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24afde1f53d95348b5a0eb19488661147285ca4dd7ed752bbc3e1c6242a304d1", size = 269779, upload-time = "2026-01-30T01:03:09.756Z" }, + { url = "https://files.pythonhosted.org/packages/20/01/7436e9ad693cebda0551203e0bf28f7669976c60ad07d6402098208476de/pytokens-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5ad948d085ed6c16413eb5fec6b3e02fa00dc29a2534f088d3302c47eb59adf9", size = 268076, upload-time = "2026-01-30T01:03:10.957Z" }, + { url = "https://files.pythonhosted.org/packages/2e/df/533c82a3c752ba13ae7ef238b7f8cdd272cf1475f03c63ac6cf3fcfb00b6/pytokens-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:3f901fe783e06e48e8cbdc82d631fca8f118333798193e026a50ce1b3757ea68", size = 103552, upload-time = "2026-01-30T01:03:12.066Z" }, + { url = "https://files.pythonhosted.org/packages/cb/dc/08b1a080372afda3cceb4f3c0a7ba2bde9d6a5241f1edb02a22a019ee147/pytokens-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8bdb9d0ce90cbf99c525e75a2fa415144fd570a1ba987380190e8b786bc6ef9b", size = 160720, upload-time = "2026-01-30T01:03:13.843Z" }, + { url = "https://files.pythonhosted.org/packages/64/0c/41ea22205da480837a700e395507e6a24425151dfb7ead73343d6e2d7ffe/pytokens-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5502408cab1cb18e128570f8d598981c68a50d0cbd7c61312a90507cd3a1276f", size = 254204, upload-time = "2026-01-30T01:03:14.886Z" }, + { url = "https://files.pythonhosted.org/packages/e0/d2/afe5c7f8607018beb99971489dbb846508f1b8f351fcefc225fcf4b2adc0/pytokens-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29d1d8fb1030af4d231789959f21821ab6325e463f0503a61d204343c9b355d1", size = 268423, upload-time = "2026-01-30T01:03:15.936Z" }, + { url = "https://files.pythonhosted.org/packages/68/d4/00ffdbd370410c04e9591da9220a68dc1693ef7499173eb3e30d06e05ed1/pytokens-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b08dd6b86058b6dc07efe9e98414f5102974716232d10f32ff39701e841c4", size = 266859, upload-time = "2026-01-30T01:03:17.458Z" }, + { url = "https://files.pythonhosted.org/packages/a7/c9/c3161313b4ca0c601eeefabd3d3b576edaa9afdefd32da97210700e47652/pytokens-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:9bd7d7f544d362576be74f9d5901a22f317efc20046efe2034dced238cbbfe78", size = 103520, upload-time = "2026-01-30T01:03:18.652Z" }, + { url = "https://files.pythonhosted.org/packages/8f/a7/b470f672e6fc5fee0a01d9e75005a0e617e162381974213a945fcd274843/pytokens-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4a14d5f5fc78ce85e426aa159489e2d5961acf0e47575e08f35584009178e321", size = 160821, upload-time = "2026-01-30T01:03:19.684Z" }, + { url = "https://files.pythonhosted.org/packages/80/98/e83a36fe8d170c911f864bfded690d2542bfcfacb9c649d11a9e6eb9dc41/pytokens-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f50fd18543be72da51dd505e2ed20d2228c74e0464e4262e4899797803d7fa", size = 254263, upload-time = "2026-01-30T01:03:20.834Z" }, + { url = "https://files.pythonhosted.org/packages/0f/95/70d7041273890f9f97a24234c00b746e8da86df462620194cef1d411ddeb/pytokens-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dc74c035f9bfca0255c1af77ddd2d6ae8419012805453e4b0e7513e17904545d", size = 268071, upload-time = "2026-01-30T01:03:21.888Z" }, + { url = "https://files.pythonhosted.org/packages/da/79/76e6d09ae19c99404656d7db9c35dfd20f2086f3eb6ecb496b5b31163bad/pytokens-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f66a6bbe741bd431f6d741e617e0f39ec7257ca1f89089593479347cc4d13324", size = 271716, upload-time = "2026-01-30T01:03:23.633Z" }, + { url = "https://files.pythonhosted.org/packages/79/37/482e55fa1602e0a7ff012661d8c946bafdc05e480ea5a32f4f7e336d4aa9/pytokens-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:b35d7e5ad269804f6697727702da3c517bb8a5228afa450ab0fa787732055fc9", size = 104539, upload-time = "2026-01-30T01:03:24.788Z" }, + { url = "https://files.pythonhosted.org/packages/30/e8/20e7db907c23f3d63b0be3b8a4fd1927f6da2395f5bcc7f72242bb963dfe/pytokens-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8fcb9ba3709ff77e77f1c7022ff11d13553f3c30299a9fe246a166903e9091eb", size = 168474, upload-time = "2026-01-30T01:03:26.428Z" }, + { url = "https://files.pythonhosted.org/packages/d6/81/88a95ee9fafdd8f5f3452107748fd04c24930d500b9aba9738f3ade642cc/pytokens-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:79fc6b8699564e1f9b521582c35435f1bd32dd06822322ec44afdeba666d8cb3", size = 290473, upload-time = "2026-01-30T01:03:27.415Z" }, + { url = "https://files.pythonhosted.org/packages/cf/35/3aa899645e29b6375b4aed9f8d21df219e7c958c4c186b465e42ee0a06bf/pytokens-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d31b97b3de0f61571a124a00ffe9a81fb9939146c122c11060725bd5aea79975", size = 303485, upload-time = "2026-01-30T01:03:28.558Z" }, + { url = "https://files.pythonhosted.org/packages/52/a0/07907b6ff512674d9b201859f7d212298c44933633c946703a20c25e9d81/pytokens-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:967cf6e3fd4adf7de8fc73cd3043754ae79c36475c1c11d514fc72cf5490094a", size = 306698, upload-time = "2026-01-30T01:03:29.653Z" }, + { url = "https://files.pythonhosted.org/packages/39/2a/cbbf9250020a4a8dd53ba83a46c097b69e5eb49dd14e708f496f548c6612/pytokens-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:584c80c24b078eec1e227079d56dc22ff755e0ba8654d8383b2c549107528918", size = 116287, upload-time = "2026-01-30T01:03:30.912Z" }, + { url = "https://files.pythonhosted.org/packages/c6/78/397db326746f0a342855b81216ae1f0a32965deccfd7c830a2dbc66d2483/pytokens-0.4.1-py3-none-any.whl", hash = "sha256:26cef14744a8385f35d0e095dc8b3a7583f6c953c2e3d269c7f82484bf5ad2de", size = 13729, upload-time = "2026-01-30T01:03:45.029Z" }, +] + +[[package]] +name = "tiflash-grafana-dashboards" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "attrs" }, + { name = "black" }, + { name = "grafanalib" }, + { name = "isort" }, +] + +[package.metadata] +requires-dist = [ + { name = "attrs" }, + { name = "black", specifier = "==26.5.1" }, + { name = "grafanalib", specifier = "==0.7.1" }, + { name = "isort", specifier = "==8.0.1" }, +] + +[[package]] +name = "tomli" +version = "2.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/22/de/48c59722572767841493b26183a0d1cc411d54fd759c5607c4590b6563a6/tomli-2.4.1.tar.gz", hash = "sha256:7c7e1a961a0b2f2472c1ac5b69affa0ae1132c39adcb67aba98568702b9cc23f", size = 17543, upload-time = "2026-03-25T20:22:03.828Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/11/db3d5885d8528263d8adc260bb2d28ebf1270b96e98f0e0268d32b8d9900/tomli-2.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f8f0fc26ec2cc2b965b7a3b87cd19c5c6b8c5e5f436b984e85f486d652285c30", size = 154704, upload-time = "2026-03-25T20:21:10.473Z" }, + { url = "https://files.pythonhosted.org/packages/6d/f7/675db52c7e46064a9aa928885a9b20f4124ecb9bc2e1ce74c9106648d202/tomli-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4ab97e64ccda8756376892c53a72bd1f964e519c77236368527f758fbc36a53a", size = 149454, upload-time = "2026-03-25T20:21:12.036Z" }, + { url = "https://files.pythonhosted.org/packages/61/71/81c50943cf953efa35bce7646caab3cf457a7d8c030b27cfb40d7235f9ee/tomli-2.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96481a5786729fd470164b47cdb3e0e58062a496f455ee41b4403be77cb5a076", size = 237561, upload-time = "2026-03-25T20:21:13.098Z" }, + { url = "https://files.pythonhosted.org/packages/48/c1/f41d9cb618acccca7df82aaf682f9b49013c9397212cb9f53219e3abac37/tomli-2.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a881ab208c0baf688221f8cecc5401bd291d67e38a1ac884d6736cbcd8247e9", size = 243824, upload-time = "2026-03-25T20:21:14.569Z" }, + { url = "https://files.pythonhosted.org/packages/22/e4/5a816ecdd1f8ca51fb756ef684b90f2780afc52fc67f987e3c61d800a46d/tomli-2.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:47149d5bd38761ac8be13a84864bf0b7b70bc051806bc3669ab1cbc56216b23c", size = 242227, upload-time = "2026-03-25T20:21:15.712Z" }, + { url = "https://files.pythonhosted.org/packages/6b/49/2b2a0ef529aa6eec245d25f0c703e020a73955ad7edf73e7f54ddc608aa5/tomli-2.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ec9bfaf3ad2df51ace80688143a6a4ebc09a248f6ff781a9945e51937008fcbc", size = 247859, upload-time = "2026-03-25T20:21:17.001Z" }, + { url = "https://files.pythonhosted.org/packages/83/bd/6c1a630eaca337e1e78c5903104f831bda934c426f9231429396ce3c3467/tomli-2.4.1-cp311-cp311-win32.whl", hash = "sha256:ff2983983d34813c1aeb0fa89091e76c3a22889ee83ab27c5eeb45100560c049", size = 97204, upload-time = "2026-03-25T20:21:18.079Z" }, + { url = "https://files.pythonhosted.org/packages/42/59/71461df1a885647e10b6bb7802d0b8e66480c61f3f43079e0dcd315b3954/tomli-2.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:5ee18d9ebdb417e384b58fe414e8d6af9f4e7a0ae761519fb50f721de398dd4e", size = 108084, upload-time = "2026-03-25T20:21:18.978Z" }, + { url = "https://files.pythonhosted.org/packages/b8/83/dceca96142499c069475b790e7913b1044c1a4337e700751f48ed723f883/tomli-2.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:c2541745709bad0264b7d4705ad453b76ccd191e64aa6f0fc66b69a293a45ece", size = 95285, upload-time = "2026-03-25T20:21:20.309Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ba/42f134a3fe2b370f555f44b1d72feebb94debcab01676bf918d0cb70e9aa/tomli-2.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c742f741d58a28940ce01d58f0ab2ea3ced8b12402f162f4d534dfe18ba1cd6a", size = 155924, upload-time = "2026-03-25T20:21:21.626Z" }, + { url = "https://files.pythonhosted.org/packages/dc/c7/62d7a17c26487ade21c5422b646110f2162f1fcc95980ef7f63e73c68f14/tomli-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7f86fd587c4ed9dd76f318225e7d9b29cfc5a9d43de44e5754db8d1128487085", size = 150018, upload-time = "2026-03-25T20:21:23.002Z" }, + { url = "https://files.pythonhosted.org/packages/5c/05/79d13d7c15f13bdef410bdd49a6485b1c37d28968314eabee452c22a7fda/tomli-2.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff18e6a727ee0ab0388507b89d1bc6a22b138d1e2fa56d1ad494586d61d2eae9", size = 244948, upload-time = "2026-03-25T20:21:24.04Z" }, + { url = "https://files.pythonhosted.org/packages/10/90/d62ce007a1c80d0b2c93e02cab211224756240884751b94ca72df8a875ca/tomli-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:136443dbd7e1dee43c68ac2694fde36b2849865fa258d39bf822c10e8068eac5", size = 253341, upload-time = "2026-03-25T20:21:25.177Z" }, + { url = "https://files.pythonhosted.org/packages/1a/7e/caf6496d60152ad4ed09282c1885cca4eea150bfd007da84aea07bcc0a3e/tomli-2.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5e262d41726bc187e69af7825504c933b6794dc3fbd5945e41a79bb14c31f585", size = 248159, upload-time = "2026-03-25T20:21:26.364Z" }, + { url = "https://files.pythonhosted.org/packages/99/e7/c6f69c3120de34bbd882c6fba7975f3d7a746e9218e56ab46a1bc4b42552/tomli-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5cb41aa38891e073ee49d55fbc7839cfdb2bc0e600add13874d048c94aadddd1", size = 253290, upload-time = "2026-03-25T20:21:27.46Z" }, + { url = "https://files.pythonhosted.org/packages/d6/2f/4a3c322f22c5c66c4b836ec58211641a4067364f5dcdd7b974b4c5da300c/tomli-2.4.1-cp312-cp312-win32.whl", hash = "sha256:da25dc3563bff5965356133435b757a795a17b17d01dbc0f42fb32447ddfd917", size = 98141, upload-time = "2026-03-25T20:21:28.492Z" }, + { url = "https://files.pythonhosted.org/packages/24/22/4daacd05391b92c55759d55eaee21e1dfaea86ce5c571f10083360adf534/tomli-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:52c8ef851d9a240f11a88c003eacb03c31fc1c9c4ec64a99a0f922b93874fda9", size = 108847, upload-time = "2026-03-25T20:21:29.386Z" }, + { url = "https://files.pythonhosted.org/packages/68/fd/70e768887666ddd9e9f5d85129e84910f2db2796f9096aa02b721a53098d/tomli-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:f758f1b9299d059cc3f6546ae2af89670cb1c4d48ea29c3cacc4fe7de3058257", size = 95088, upload-time = "2026-03-25T20:21:30.677Z" }, + { url = "https://files.pythonhosted.org/packages/07/06/b823a7e818c756d9a7123ba2cda7d07bc2dd32835648d1a7b7b7a05d848d/tomli-2.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:36d2bd2ad5fb9eaddba5226aa02c8ec3fa4f192631e347b3ed28186d43be6b54", size = 155866, upload-time = "2026-03-25T20:21:31.65Z" }, + { url = "https://files.pythonhosted.org/packages/14/6f/12645cf7f08e1a20c7eb8c297c6f11d31c1b50f316a7e7e1e1de6e2e7b7e/tomli-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:eb0dc4e38e6a1fd579e5d50369aa2e10acfc9cace504579b2faabb478e76941a", size = 149887, upload-time = "2026-03-25T20:21:33.028Z" }, + { url = "https://files.pythonhosted.org/packages/5c/e0/90637574e5e7212c09099c67ad349b04ec4d6020324539297b634a0192b0/tomli-2.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c7f2c7f2b9ca6bdeef8f0fa897f8e05085923eb091721675170254cbc5b02897", size = 243704, upload-time = "2026-03-25T20:21:34.51Z" }, + { url = "https://files.pythonhosted.org/packages/10/8f/d3ddb16c5a4befdf31a23307f72828686ab2096f068eaf56631e136c1fdd/tomli-2.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f3c6818a1a86dd6dca7ddcaaf76947d5ba31aecc28cb1b67009a5877c9a64f3f", size = 251628, upload-time = "2026-03-25T20:21:36.012Z" }, + { url = "https://files.pythonhosted.org/packages/e3/f1/dbeeb9116715abee2485bf0a12d07a8f31af94d71608c171c45f64c0469d/tomli-2.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d312ef37c91508b0ab2cee7da26ec0b3ed2f03ce12bd87a588d771ae15dcf82d", size = 247180, upload-time = "2026-03-25T20:21:37.136Z" }, + { url = "https://files.pythonhosted.org/packages/d3/74/16336ffd19ed4da28a70959f92f506233bd7cfc2332b20bdb01591e8b1d1/tomli-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51529d40e3ca50046d7606fa99ce3956a617f9b36380da3b7f0dd3dd28e68cb5", size = 251674, upload-time = "2026-03-25T20:21:38.298Z" }, + { url = "https://files.pythonhosted.org/packages/16/f9/229fa3434c590ddf6c0aa9af64d3af4b752540686cace29e6281e3458469/tomli-2.4.1-cp313-cp313-win32.whl", hash = "sha256:2190f2e9dd7508d2a90ded5ed369255980a1bcdd58e52f7fe24b8162bf9fedbd", size = 97976, upload-time = "2026-03-25T20:21:39.316Z" }, + { url = "https://files.pythonhosted.org/packages/6a/1e/71dfd96bcc1c775420cb8befe7a9d35f2e5b1309798f009dca17b7708c1e/tomli-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:8d65a2fbf9d2f8352685bc1364177ee3923d6baf5e7f43ea4959d7d8bc326a36", size = 108755, upload-time = "2026-03-25T20:21:40.248Z" }, + { url = "https://files.pythonhosted.org/packages/83/7a/d34f422a021d62420b78f5c538e5b102f62bea616d1d75a13f0a88acb04a/tomli-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:4b605484e43cdc43f0954ddae319fb75f04cc10dd80d830540060ee7cd0243cd", size = 95265, upload-time = "2026-03-25T20:21:41.219Z" }, + { url = "https://files.pythonhosted.org/packages/3c/fb/9a5c8d27dbab540869f7c1f8eb0abb3244189ce780ba9cd73f3770662072/tomli-2.4.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fd0409a3653af6c147209d267a0e4243f0ae46b011aa978b1080359fddc9b6cf", size = 155726, upload-time = "2026-03-25T20:21:42.23Z" }, + { url = "https://files.pythonhosted.org/packages/62/05/d2f816630cc771ad836af54f5001f47a6f611d2d39535364f148b6a92d6b/tomli-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a120733b01c45e9a0c34aeef92bf0cf1d56cfe81ed9d47d562f9ed591a9828ac", size = 149859, upload-time = "2026-03-25T20:21:43.386Z" }, + { url = "https://files.pythonhosted.org/packages/ce/48/66341bdb858ad9bd0ceab5a86f90eddab127cf8b046418009f2125630ecb/tomli-2.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:559db847dc486944896521f68d8190be1c9e719fced785720d2216fe7022b662", size = 244713, upload-time = "2026-03-25T20:21:44.474Z" }, + { url = "https://files.pythonhosted.org/packages/df/6d/c5fad00d82b3c7a3ab6189bd4b10e60466f22cfe8a08a9394185c8a8111c/tomli-2.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:01f520d4f53ef97964a240a035ec2a869fe1a37dde002b57ebc4417a27ccd853", size = 252084, upload-time = "2026-03-25T20:21:45.62Z" }, + { url = "https://files.pythonhosted.org/packages/00/71/3a69e86f3eafe8c7a59d008d245888051005bd657760e96d5fbfb0b740c2/tomli-2.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7f94b27a62cfad8496c8d2513e1a222dd446f095fca8987fceef261225538a15", size = 247973, upload-time = "2026-03-25T20:21:46.937Z" }, + { url = "https://files.pythonhosted.org/packages/67/50/361e986652847fec4bd5e4a0208752fbe64689c603c7ae5ea7cb16b1c0ca/tomli-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ede3e6487c5ef5d28634ba3f31f989030ad6af71edfb0055cbbd14189ff240ba", size = 256223, upload-time = "2026-03-25T20:21:48.467Z" }, + { url = "https://files.pythonhosted.org/packages/8c/9a/b4173689a9203472e5467217e0154b00e260621caa227b6fa01feab16998/tomli-2.4.1-cp314-cp314-win32.whl", hash = "sha256:3d48a93ee1c9b79c04bb38772ee1b64dcf18ff43085896ea460ca8dec96f35f6", size = 98973, upload-time = "2026-03-25T20:21:49.526Z" }, + { url = "https://files.pythonhosted.org/packages/14/58/640ac93bf230cd27d002462c9af0d837779f8773bc03dee06b5835208214/tomli-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:88dceee75c2c63af144e456745e10101eb67361050196b0b6af5d717254dddf7", size = 109082, upload-time = "2026-03-25T20:21:50.506Z" }, + { url = "https://files.pythonhosted.org/packages/d5/2f/702d5e05b227401c1068f0d386d79a589bb12bf64c3d2c72ce0631e3bc49/tomli-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:b8c198f8c1805dc42708689ed6864951fd2494f924149d3e4bce7710f8eb5232", size = 96490, upload-time = "2026-03-25T20:21:51.474Z" }, + { url = "https://files.pythonhosted.org/packages/45/4b/b877b05c8ba62927d9865dd980e34a755de541eb65fffba52b4cc495d4d2/tomli-2.4.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:d4d8fe59808a54658fcc0160ecfb1b30f9089906c50b23bcb4c69eddc19ec2b4", size = 164263, upload-time = "2026-03-25T20:21:52.543Z" }, + { url = "https://files.pythonhosted.org/packages/24/79/6ab420d37a270b89f7195dec5448f79400d9e9c1826df982f3f8e97b24fd/tomli-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7008df2e7655c495dd12d2a4ad038ff878d4ca4b81fccaf82b714e07eae4402c", size = 160736, upload-time = "2026-03-25T20:21:53.674Z" }, + { url = "https://files.pythonhosted.org/packages/02/e0/3630057d8eb170310785723ed5adcdfb7d50cb7e6455f85ba8a3deed642b/tomli-2.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1d8591993e228b0c930c4bb0db464bdad97b3289fb981255d6c9a41aedc84b2d", size = 270717, upload-time = "2026-03-25T20:21:55.129Z" }, + { url = "https://files.pythonhosted.org/packages/7a/b4/1613716072e544d1a7891f548d8f9ec6ce2faf42ca65acae01d76ea06bb0/tomli-2.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:734e20b57ba95624ecf1841e72b53f6e186355e216e5412de414e3c51e5e3c41", size = 278461, upload-time = "2026-03-25T20:21:56.228Z" }, + { url = "https://files.pythonhosted.org/packages/05/38/30f541baf6a3f6df77b3df16b01ba319221389e2da59427e221ef417ac0c/tomli-2.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8a650c2dbafa08d42e51ba0b62740dae4ecb9338eefa093aa5c78ceb546fcd5c", size = 274855, upload-time = "2026-03-25T20:21:57.653Z" }, + { url = "https://files.pythonhosted.org/packages/77/a3/ec9dd4fd2c38e98de34223b995a3b34813e6bdadf86c75314c928350ed14/tomli-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:504aa796fe0569bb43171066009ead363de03675276d2d121ac1a4572397870f", size = 283144, upload-time = "2026-03-25T20:21:59.089Z" }, + { url = "https://files.pythonhosted.org/packages/ef/be/605a6261cac79fba2ec0c9827e986e00323a1945700969b8ee0b30d85453/tomli-2.4.1-cp314-cp314t-win32.whl", hash = "sha256:b1d22e6e9387bf4739fbe23bfa80e93f6b0373a7f1b96c6227c32bef95a4d7a8", size = 108683, upload-time = "2026-03-25T20:22:00.214Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/da524626d3b9cc40c168a13da8335fe1c51be12c0a63685cc6db7308daae/tomli-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:2c1c351919aca02858f740c6d33adea0c5deea37f9ecca1cc1ef9e884a619d26", size = 121196, upload-time = "2026-03-25T20:22:01.169Z" }, + { url = "https://files.pythonhosted.org/packages/5a/cd/e80b62269fc78fc36c9af5a6b89c835baa8af28ff5ad28c7028d60860320/tomli-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:eab21f45c7f66c13f2a9e0e1535309cee140182a9cdae1e041d02e47291e8396", size = 100393, upload-time = "2026-03-25T20:22:02.137Z" }, + { url = "https://files.pythonhosted.org/packages/7b/61/cceae43728b7de99d9b847560c262873a1f6c98202171fd5ed62640b494b/tomli-2.4.1-py3-none-any.whl", hash = "sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe", size = 14583, upload-time = "2026-03-25T20:22:03.012Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +]