High cardinality of metrics

Checks

I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
I am using charts that are officially provided

Controller Version

0.7.0

Deployment Method

Helm

Checks

This isn't a question or user support case (For Q&A and community support, go to Discussions).
I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

enable metrics to controller by using:

metrics:
controllerManagerAddr: ":8080"
listenerAddr: ":8080"
listenerEndpoint: "/metrics"

you need to have enough high traffic build environment that it will build a lot from different repos and prs

Describe the bug

Listeners contains metrics which is great. However, the cardinality of these metrics are just something that is not going to work. There needs to be way to disable high cardinality labels from the metrics.

so just during one hour we got over 15k new timeseries to our prometheus which is going to explode if we keep these enabled even 12 hours.

Describe the expected behavior

The expected behaviour is that high cardinality labels is REMOVED from the metrics. Also metrics buckets should be configurable in case of bucket metrics.

Lets take the worst metric gha_job_execution_duration_seconds_bucket

it do have job_workflow_ref which is basically almost always unique, which means that it will always create new timeseries to prometheus which is really expensive. Worth of thinking is even job_name needed. I would like to disable both of these labels. Also number of the default buckets are something which is going to explode our prometheus.

Additional Context

replicaCount: 2

flags:
  logLevel: "debug"

metrics:
  controllerManagerAddr: ":8080"
  listenerAddr: ":8080"
  listenerEndpoint: "/metrics"

Controller Logs

not relevant

Runner Pod Logs

not relevant

I have now disabled high cardinality metrics with relabelings. However, I would like to see removal of job_name and job_workflow_ref from all of these metrics even possibility to configure that. These might work in environment which do have like 10 builds per day but we do have more than thousand per hour.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: listeners
  labels:
    app.kubernetes.io/part-of: gha-runner-scale-set
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: gha-runner-scale-set
  podMetricsEndpoints:
  - port: metrics
    relabelings:
    - action: drop
      regex: 'gha_job_(execution|startup)_duration_seconds'
    - action: drop
      regex: 'gha_completed_jobs_total|gha_started_jobs_total'

The labels on both gha_job_execution_duration_seconds and gha_job_startup_duration_seconds metrics mean that a new bucket is created for every run on every job, this means that every bucket will only ever contain a 0 or a 1. You cannot get meaningful information out of these metrics.

Prometheus is unable to aggregate metrics before applying rate() on them to produce histograms, so with the current layout of these metrics it is impossible to produce a histogram of startup or execution durations.

Information should be put into buckets based on job_name, organisation, and repository only. Highly unique labels such as runner_id, runner_name, and job_workflow_ref should be removed.

Also, it's very likely to get scrape errors after many builds:

"http://10.2.0.23:8080/metrics" exceeds -promscrape.maxScrapeSize=16777216

@nikola-jokic

@nikola-jokic is there any possibility that this could end under development?

For my use case, this is definitely important metrics to expose. I'd love to have the metrics to be able to monitor the workflows properly, but the current setup makes it close to impossible.

If we remove the highly unique labels, and add a label for the name that would probably solve everything.

Hi, any updates on this ? we're experiencing the same issue and had to turn off the metrics altogether because opentelemetry agents couldn't handle the data.

This may have been because I am using grafana-alloy to do the scrapes, but in order to remove the metrics I had to use this relabeling config:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: gha-runner-gha-rs-listeners
  labels:
    release: grafana-alloy
spec:
  selector:
    matchLabels:
      "app.kubernetes.io/component": runner-scale-set-listener
  podMetricsEndpoints:
    - port: metrics
      metricRelabelings:
        - action: drop
          regex: "gha_job_(execution|startup)_duration_seconds"
          sourceLabels: ["__name__"]
        - action: drop
          regex: "gha_completed_jobs_total|gha_started_jobs_total"
          sourceLabels: ["__name__"]

This uses the metricsRelabelings from: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#podmetricsendpoint

The positive is these relabeling rules happen before ingestion

metricRelabelings configures the relabeling rules to apply to the samples before ingestion.

We're suffering from this as well, with Datadog scraping the metrics endpoint. The payload returned just grows infinitely the longer the listener runs with all the jobs we run constantly. We had to significantly increase the memory allocated to our datadog agent pods just so that it had enough memory to load and process the returned metrics without being oomkilled, and even then we still hit the cap of max metrics the agent will ingest within a few hours of the listener pod starting.

Even aside from the metrics cardinality problem, I notice the memory usage of the listener pod increases steadily over time, possibly related to keeping track of all these metrics. I can see the memory usage grow from ~20mb to over 700mb for one listener pod over a couple weeks.

Digging into this a bit more, it seems like this is related to the deliberate decision on the part of prometheus/client_golang to never expire old metrics. (I had hoped maybe there would be an optional flag to expire these similar to statsd's config.deleteCounters but that doesn't appear to be an option.)

Which suggests the only options are 1) don't include high cardinality tags, or 2) explicitly implement a solution to Unregister these metrics after they're no longer being updated. I suppose there is 3) constantly restart the controller but that's untenable as we'd have to restart it every few hours.

We're facing the same issue using only ephemeral runners.

#3556 would fix the runner_name and runner_id cardinality but it would be nice to customize this as we don't even care about job_workflow_ref and the branch, just want to monitor count/duration of repo+job over time :-)

I have modified the Datadog openmetrics annotation to ignore certain labels:

"exclude_labels": [
  "runner_id",
  "runner_name",
  "enterprise",
  "organization",
  "job_result",
  "job_workflow_ref",
  "event_name"
]

Unfortunately this doesn't fix the maximum metric scrape count of 2000 and the only way to fix this is to restart the listener pods...

Also running into issues. This controller alone was consuming over 8 million series in our Prometheus instance, and we've only migrated 5-10% of our users to GHA. Definitely would suggest either an option to toggle details monitoring, or drop most of the extremely-high cardinality labels (runner name, id, job workflow ref, etc). We'd like to use those metrics but for now, we have to drop them due to the monitoring burden.

Would love to see GHA runners be compliant in some form with the OTel semantic conventions being worked under https://github.com/open-telemetry/community/blob/main/projects/ci-cd.md

Part of that is deciding what metrics and what attributes make sense, compared to more high-granularity data provided by traces

@danielgblanco

As @danielgblanco mentioned, would love to see issues like this help inform the semantic conventions being driven. We hold weekly meetings every Thursday at 06:00 PT which can be found on the OpenTelemetry Calendar. In working with the larger community, we've already established some key CICD and VCS metric metric semantics. There's also other host specific metrics that may apply more generally.

Aside from that, there's a key area of value that can't be retrieved through metrics, but does enable additional metrics to be derived, and that's tracing. There's active on-going work to derive traces from GitHub workflow action runs. There's also active work being done on an environment variable specification to enable native tracing to be emitted compliantly from the pipelines directly.

Navigation Menu

Repository navigation

High cardinality of metrics #3153

Checks

Controller Version

Deployment Method

Checks

To Reproduce

Describe the bug

Describe the expected behavior

Additional Context

Controller Logs

Runner Pod Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

High cardinality of metrics #3153

Description

Checks

Controller Version

Deployment Method

Checks

To Reproduce

Describe the bug

Describe the expected behavior

Additional Context

Controller Logs

Runner Pod Logs

Activity

zetaab commented on Dec 14, 2023

thomassandslyst commented on May 28, 2024

notz commented on Jun 11, 2024

zetaab commented on Jun 11, 2024

realmunk commented on Jul 17, 2024

th-le commented on Aug 28, 2024

alecrajeev commented on Oct 3, 2024

mikespharss commented on Oct 9, 2024

mikespharss commented on Oct 10, 2024

yvesjans commented on Oct 24, 2024

lrstanley commented on Feb 10, 2025

danielgblanco commented on Feb 11, 2025

adrielp commented on Feb 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions