HDDS-15396. Grafana dashboard for disk balancer by navinko · Pull Request #10609 · apache/ozone

navinko · 2026-06-25T18:45:18Z

What changes were proposed in this pull request?

Created brand new grafana dashboard for disk balancer.
Used already exposed prometheus matrices for disk balancer.

Please describe your PR in detail:

Used volume_info_metrics + below metrices in Grafana dashboard for tracking

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15396

How was this patch tested?

Simulated skew disk for datanode in my mac-> Triggered disk balancer through CLI using docker-compose -> Captured dashboard screen shot and validated panel
Used same json "Ozone - Disk Balancer.json" for current PR

Screen shots for references:

jojochuang · 2026-06-26T21:58:54Z

The metric name needs to update, because the volume info metric class produces metrics names like this:

volume_info_metrics_data_12_hadoop_ozone_datanode_data_vadiraja_reserved{context="ozone",storagetype="DISK",datanodeuuid="8f6f944b-77b3-4a95-91da-c9d9b51d773b",volumetype="DATA_VOLUME",storagedirectory="/data/12/hadoop-ozone/datanode/data/vadiraja/hdds",volumestate="NORMAL",hostname="drc6-oz-worker07.sjc.cloudera.com"}

where the metric name is appended with the volume root directory path.

Gargi-jais11

Thanks @navinko for working on the patch.
Left review comments below

navinko · 2026-07-18T17:15:49Z

Thanks @Gargi-jais11 for the review. Updated the changes and tried addressing suggestions.
Few more screenshot based on local testing.

Gargi-jais11

Thanks @navinko for updating the patch.
Just few more fixes to be done.

Gargi-jais11 · 2026-07-22T07:07:58Z

+      }
+    ]
+  },
+  "description": "",


Suggested change

"description": "",

"description": "Monitors Ozone Disk Balancer progress, throughput, and idle state per datanode.",

Gargi-jais11 · 2026-07-22T07:09:33Z

+            "uid": "${datasource}"
+          },
+          "expr": "sum({__name__=~\"volume_info_metrics_.*_ozone_used\", instance=~\"$datanode\"}) by (instance, storagedirectory)",
+          "legendFormat": "{{storagedirectory}}",


With multiple datanodes selected, duplicate paths from different hosts are indistinguishable.

Suggested change

"legendFormat": "{{storagedirectory}}",

"legendFormat": "{{instance}} {{storagedirectory}}",

Gargi-jais11 · 2026-07-22T07:13:34Z

+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "expr": "rate(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1m])",


[1m] at a typical 15s Prometheus scrape interval gives only 4 samples. This makes "Active Balancing Throughput Speed" extremely spiky. Use [$__rate_interval] (Grafana auto-calculates the appropriate window) or at minimum [5m].

Suggested change

"expr": "rate(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1m])",

"expr": "rate(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[$__rate_interval])",

Gargi-jais11 · 2026-07-22T07:15:33Z

+              }
+            ]
+          },
+          "unit": "m"


If diskbalancer is in runnign state but not actively balancing than showing "0mins is ambigous"

Suggested change

"unit": "m"

"unit": "m",

"noValue": "N/A"

Gargi-jais11 · 2026-07-22T07:19:46Z

+          "expr": "{__name__=~\"disk_balancer_service_metrics_move_success_time_avg_time|disk_balancer_service_metrics_move_failure_time_avg_time\", instance=~\"$datanode\"}",
+          "legendFormat": "{{__name__}}",
+          "refId": "A"
+        }
+      ],


better to break it in this way.

Suggested change

"expr": "{__name__=~\"disk_balancer_service_metrics_move_success_time_avg_time|disk_balancer_service_metrics_move_failure_time_avg_time\", instance=~\"$datanode\"}",

"legendFormat": "{{__name__}}",

"refId": "A"

}

],

"expr": "disk_balancer_service_metrics_move_success_time_avg_time{instance=~\"$datanode\"}",

"legendFormat": "{{instance}} success avg latency",

"refId": "A"

},

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

},

"expr": "disk_balancer_service_metrics_move_failure_time_avg_time{instance=~\"$datanode\"}",

"legendFormat": "{{instance}} failure avg latency",

"refId": "B"

},

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

},

"expr": "disk_balancer_service_metrics_move_success_time_num_ops{instance=~\"$datanode\"}",

"legendFormat": "{{instance}} moves/interval",

"refId": "C"

}

],

Gargi-jais11 · 2026-07-22T07:23:25Z

+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "expr": "max_over_time(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1h])",


Same the counter is cumulative:

Suggested change

"expr": "max_over_time(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1h])",

"expr": "disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}",

It's giving bad data : invalid parameter hence used [1h]

Gargi-jais11 · 2026-07-22T07:24:10Z

+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "expr": "max_over_time(disk_balancer_service_metrics_failure_count{instance=~\"$datanode\"}[1h])",


Same cumulative counter.

Suggested change

"expr": "max_over_time(disk_balancer_service_metrics_failure_count{instance=~\"$datanode\"}[1h])",

"expr": "(disk_balancer_service_metrics_failure_count{instance=~\"$datanode\"})",

It's giving bad data : invalid parameter hence used [1h]

Gargi-jais11 · 2026-07-22T07:28:45Z

+      "title": "Balancer Idle Reason",
+      "type": "piechart"
+    }
+  ],


There is one more metrics which shows how many times a balancer is iterating through the loop if balancer is running.
Roughly I created this locally u can change it accordingly.

{ "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "Rate of disk balancer scheduler loop iterations per second. A non-zero value confirms the balancer process is alive and iterating. If this drops to zero while the balancer is expected to run, the service may have stalled or been stopped.", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "loops/sec", "axisPlacement": "auto", "barAlignment": 0, "barWidthFactor": 0.6, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "showValues": false, "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": 0 }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 5, "w": 24, "x": 0, "y": 20 }, "id": 16, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "hideZeros": false, "mode": "multi", "sort": "none" } }, "pluginVersion": "13.0.2", "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "expr": "rate(disk_balancer_service_metrics_running_loop_count{instance=~\"$datanode\"}[$__rate_interval])", "legendFormat": "{{instance}} loop rate", "refId": "A" } ], "title": "Balancer Heartbeat \u2014 Scheduler Loop Rate", "type": "timeseries" }

Thanks for sharing , Created new panel -

Gargi-jais11 · 2026-07-22T07:29:28Z

+  "preload": false,
+  "refresh": "5s",
+  "schemaVersion": 42,
+  "tags": [],


Let's give tag:

Suggested change

"tags": [],

"tags": [

"ozone",

"datanode",

"disk-balancer"

],

Added tag as suggested.

Gargi-jais11 · 2026-07-22T07:32:37Z

+          "expr": "sum(increase(disk_balancer_service_metrics_idle_loop_no_available_volume_pair_count{instance=~\"$datanode\"}[15m]))",
+          "instant": true,


Suggested change

"expr": "sum(increase(disk_balancer_service_metrics_idle_loop_no_available_volume_pair_count{instance=~\"$datanode\"}[15m]))",

"instant": true,

"expr": "sum(increase(disk_balancer_service_metrics_idle_loop_no_available_volume_pair_count{instance=~\"$datanode\"}[$__range]))",

"instant": false,

navinko · 2026-08-06T18:36:41Z

Thanks @Gargi-jais11 for the review . Updated the json with addressed review comments.

HDDS-15396. Added new grafana dashboard for disk balancer

38bde60

jojochuang reviewed Jun 26, 2026

View reviewed changes

Comment thread hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk Balancer.json Outdated

Comment thread hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk Balancer.json Outdated

HDDS-15396. Addressed review comments

a912e88

navinko requested a review from jojochuang June 28, 2026 05:16

ivandika3 requested a review from Gargi-jais11 July 7, 2026 02:42

Gargi-jais11 requested a review from ChenSammi July 15, 2026 06:00

Gargi-jais11 reviewed Jul 15, 2026

View reviewed changes

Comment thread hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk Balancer.json Outdated

Gargi-jais11 reviewed Jul 15, 2026

View reviewed changes

Comment thread hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk Balancer.json Outdated

Gargi-jais11 reviewed Jul 15, 2026

View reviewed changes

Comment thread hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone - Disk Balancer.json

HDDS-15396. Addressed review comments

bbedcf7

navinko requested a review from Gargi-jais11 July 18, 2026 17:16

Merge branch 'apache:master' into HDDS-15396

f18488b

Gargi-jais11 reviewed Jul 22, 2026

View reviewed changes

navinko marked this pull request as draft July 22, 2026 17:00

navinko added 2 commits August 6, 2026 19:10

Merge branch 'apache:master' into HDDS-15396

da899fa

HDDS-15396. Addressed review comments

47a7093

navinko marked this pull request as ready for review August 6, 2026 18:38

navinko requested a review from Gargi-jais11 August 6, 2026 18:38

	"description": "",
	"description": "Monitors Ozone Disk Balancer progress, throughput, and idle state per datanode.",

	"legendFormat": "{{storagedirectory}}",
	"legendFormat": "{{instance}} {{storagedirectory}}",

	"expr": "rate(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1m])",
	"expr": "rate(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[$__rate_interval])",

-          "expr": "{__name__=~\"disk_balancer_service_metrics_move_success_time_avg_time|disk_balancer_service_metrics_move_failure_time_avg_time\", instance=~\"$datanode\"}",
-          "legendFormat": "{{__name__}}",
-          "refId": "A"
-        }
-      ],
+         "expr": "disk_balancer_service_metrics_move_success_time_avg_time{instance=~\"$datanode\"}",
+          "legendFormat": "{{instance}} success avg latency",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "expr": "disk_balancer_service_metrics_move_failure_time_avg_time{instance=~\"$datanode\"}",
+          "legendFormat": "{{instance}} failure avg latency",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "expr": "disk_balancer_service_metrics_move_success_time_num_ops{instance=~\"$datanode\"}",
+          "legendFormat": "{{instance}} moves/interval",
+          "refId": "C"
+        }
+      ],

	"expr": "max_over_time(disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}[1h])",
	"expr": "disk_balancer_service_metrics_success_bytes{instance=~\"$datanode\"}",

	"expr": "max_over_time(disk_balancer_service_metrics_failure_count{instance=~\"$datanode\"}[1h])",
	"expr": "(disk_balancer_service_metrics_failure_count{instance=~\"$datanode\"})",

		"expr": "sum(increase(disk_balancer_service_metrics_idle_loop_no_available_volume_pair_count{instance=~\"$datanode\"}[15m]))",
		"instant": true,

Uh oh!

Conversation

navinko commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Please describe your PR in detail:

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jojochuang commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

navinko commented Jul 18, 2026

Uh oh!

Gargi-jais11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinko Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navinko commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

navinko commented Jun 25, 2026 •

edited

Loading

navinko Aug 6, 2026 •

edited

Loading

navinko commented Aug 6, 2026 •

edited

Loading