Skip to content

Loki: option to disable stats.grafana.org telemetry #8210

Closed
Feature
Closed
Loki: option to disable stats.grafana.org telemetry#8210
Feature
Labels
verifiedAll test cases were verified successfully
Milestone
NethServer 8.11

Description

@DavidePrincipi
DavidePrincipi
opened on Oct 8, 2026

Loki, the log storage app of NethServer 8, periodically sends anonymous usage statistics to stats.grafana.org. When the network uses a DNS (Domain Name System) blocker like Pi-hole that blocks this name, Loki keeps retrying the failed lookup at a very high rate, and Pi-hole users see thousands of queries for stats.grafana.org coming from the NethServer 8 node.

Usage statistics are already disabled in the Alloy log collector (the promtail.service unit) by NethServer/ns8-core@d839475 (NethServer/ns8-core#1230). Loki still sends them, and an upstream bug (grafana/loki#10648) reports that Loki may contact stats.grafana.org even when usage reporting is disabled in its configuration.

The purpose of this feature is to let the administrator stop this traffic with a manual configuration. The default behavior does not change: without manual configuration, Loki keeps sending usage statistics as it does now.

Proposed solution

Add two environment variables to the Loki app:

  • LOKI_REPORTING_ENABLED (default true): when set to false, disables the Loki usage reporting (analytics.reporting_enabled in the Loki configuration)
  • PODMAN_POD_OPTS (default empty): extra options for the podman pod create command of the Loki pod. For example, --add-host=stats.grafana.org:127.0.0.1 resolves stats.grafana.org to the loopback address inside the pod, so connection attempts fail immediately without leaving the node. This works around the upstream bug.

Document in the README how to set the variables with runagent and restart the service.

Alternative solutions

  • Add a fake stats.grafana.org record to /etc/hosts on the node, as suggested in the community. It affects the whole node, not only Loki.
  • Point stats.grafana.org to an unreachable address (for example a documentation range). Each connection attempt would hang until a timeout instead of failing immediately, so the loopback address is preferred.
  • Disable usage reporting by default. Not proposed for now, to keep the current behavior unchanged.

Additional context

Affected component: Loki 1.5.1

See also

Activity

  1. DavidePrincipi
    added the
    Bug
    issue type on Oct 8, 2026
  2. DavidePrincipi
    added this to NethServeron Oct 8, 2026
  3. DavidePrincipi
    added this to the NethServer 8.10 milestone on Oct 8, 2026
  4. github-project-automation
    moved this to ToDo in NethServeron Oct 8, 2026
  5. DavidePrincipi
    modified the milestones: NethServer 8.10, NethServer 8.11 on Oct 8, 2026
  6. DavidePrincipi
    changed the title [-]Loki floods DNS with stats.grafana.org lookups[/-] [+]Loki: option to disable stats.grafana.org telemetry[/+] on Oct 8, 2026
  7. DavidePrincipi
    changed the issue type from
    Bug
    to
    Feature
    on Oct 8, 2026

  8. DavidePrincipi commented on Oct 8, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 8, 2026
    MemberAuthor

    QA

    Test images: ghcr.io/nethserver/loki:1.5.2-dev.1 and ghcr.io/nethserver/metrics:1.5.1-dev.1. The commands assume the default instances loki1 and metrics1.

    Loki

    1. Update the module and check that logs are still collected: 
      update-module ghcr.io/nethserver/loki:1.5.2-dev.1 loki1
    2. Disable usage reporting: 
      runagent -m loki1 python3 -c 'import agent ; agent.set_env("LOKI_REPORTING_ENABLED", "false")'
runagent -m loki1 systemctl --user restart loki.service
runagent -m loki1 podman inspect loki-server --format '{{.Config.Env}}' | grep -o 'LOKI_REPORTING_ENABLED=[a-z]*'
      Expected: LOKI_REPORTING_ENABLED=false, and Loki starts normally.
    3. Resolve stats.grafana.org to the loopback address inside the pod: 
      runagent -m loki1 python3 -c 'import agent ; agent.set_env("PODMAN_POD_OPTS", "--add-host=stats.grafana.org:127.0.0.1")'
runagent -m loki1 systemctl --user restart loki.service
runagent -m loki1 sh -c 'grep grafana $(podman inspect loki-server --format {{.HostsPath}})'
      Expected: 127.0.0.1 stats.grafana.org.
    4. Revert, and check that the defaults come back: 
      runagent -m loki1 python3 -c 'import agent ; agent.munset_env(["LOKI_REPORTING_ENABLED", "PODMAN_POD_OPTS"])'
runagent -m loki1 systemctl --user restart loki.service

    Metrics (Grafana)

    Grafana must be enabled (grafana_path set).

    1. Update the module: 
      update-module ghcr.io/nethserver/metrics:1.5.1-dev.1 metrics1
    2. Update checks must be off by default: 
      runagent -m metrics1 podman exec grafana env | grep GF_ANALYTICS
      Expected: GF_ANALYTICS_CHECK_FOR_UPDATES=false and GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false. Dashboards load as before.
    3. Disable usage reporting: 
      runagent -m metrics1 python3 -c 'import agent ; agent.set_env("PODMAN_RUN_OPTS", "-e GF_ANALYTICS_REPORTING_ENABLED=false")'
runagent -m metrics1 systemctl --user restart grafana.service
runagent -m metrics1 podman exec grafana env | grep GF_ANALYTICS
      Expected: GF_ANALYTICS_REPORTING_ENABLED=false is listed too.
    4. Revert: 
      runagent -m metrics1 python3 -c 'import agent ; agent.unset_env("PODMAN_RUN_OPTS")'
runagent -m metrics1 systemctl --user restart grafana.service

    Note: we couldn't reproduce the DNS query flood in the lab. With a simulated Pi-hole, Loki made no lookups for stats.grafana.org in 14 minutes after a restart. These steps check that the settings are applied. If you have a node behind Pi-hole, compare its query counts before and after the change.

  9. DavidePrincipi
    added
    testingPackages are available from testing repositories
    on Oct 8, 2026

  10. DavidePrincipi commented on Oct 8, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 8, 2026
    MemberAuthor

    Already released https://github.com/NethServer/ns8-metrics/releases/tag/1.5.1

  11. DavidePrincipi
    self-assigned this
    on Oct 9, 2026

  12. DavidePrincipi commented on Oct 14, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 14, 2026
    MemberAuthor

    Test results

    Test setup: a single-node cluster (Rocky Linux 9) with the ns8-dnsmasq module acting like Pi-hole. Its DNS records answer 0.0.0.0 and :: for stats.grafana.org and grafana.com (Pi-hole's default blocking mode), and log-queries records every request. The node itself uses dnsmasq as its DNS server.

    Component Configuration stats.grafana.org DNS queries
    Loki 3.7.6 (loki 1.5.1) defaults about 600 per hour, starting about 2h20m after Loki starts and never stopping
    Loki 3.7.6 (loki 1.5.1) defaults, Loki stopped for 3 minutes 0 while stopped (30 in the 3 minutes before), so Loki is the source
    Loki 3.7.8 (loki 1.5.2-dev.1) LOKI_REPORTING_ENABLED=false 0 in 7 hours
    Loki 3.7.8 (loki 1.5.2-dev.1) reporting enabled, PODMAN_POD_OPTS=--add-host=stats.grafana.org:127.0.0.1 0 in 18.5 hours
    Grafana 13.2.3 (metrics 1.5.1) defaults one A + AAAA pair when the daily report is due, no retries
    Grafana 13.2.3 (metrics 1.5.1) PODMAN_RUN_OPTS=-e GF_ANALYTICS_REPORTING_ENABLED=false 0

    Notes:

    • Loki with defaults: the flood comes from the usage report retry loop. Gaps between lookups grow from 2 to about 15 seconds, then the cycle starts again after 30-40 seconds.
    • LOKI_REPORTING_ENABLED=false: Loki's /config endpoint shows analytics.reporting_enabled: false. With Loki 3.7.8, the upstream issue Loki Reports to stats.grafana.org despite analytics.reporting_enabled: false grafana/loki#10648 (reports sent despite this setting) did not show up during the test.
    • --add-host: Loki keeps retrying, but only inside its pod. A traffic capture on the pod loopback interface showed about one connection to 127.0.0.1:443 every 12 seconds, each refused immediately, with no DNS query and no traffic leaving the node.
    • Grafana: usage statistics are sent once every 24 hours, and a failed report is not retried, so Grafana does not flood DNS. To avoid waiting a day, the test moved Grafana's stored "last sent" time back by 24 hours before each restart. With reporting disabled, the daily timer still fires but nothing is sent. Pointing the blocked name to 0.0.0.0 makes Grafana, which uses host networking, connect to the node's own Traefik on port 443 and fail the certificate check, which is harmless.
    • Grafana update checks: they are disabled by default in metrics 1.5.1. Grafana still made about 36 grafana.com lookups on every start, from its plugin preinstall feature. This is addressed separately in feat: Grafana image with bundled plugins only ns8-metrics#99.

  13. DavidePrincipi commented on Oct 14, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 14, 2026
    MemberAuthor

    QA: metrics 1.6.0-dev.2

    Test image: ghcr.io/nethserver/metrics:1.6.0-dev.2 (NethServer/ns8-metrics#99). Grafana now runs from ghcr.io/nethserver/metrics-grafana, which bundles Logs Drilldown and Metrics Drilldown, and never downloads plugins.

    Setup: metrics1 on 1.5.x with Grafana enabled (grafana_path set) and internet access, so Grafana has already downloaded its plugins:

    runagent -m metrics1 podman exec grafana ls /var/lib/grafana/plugins

    Expected: a list of plugins (17 on a fresh 1.5.1), e.g. grafana-lokiexplore-app, loki, prometheus.

    Steps

    1. Update the module: 
      update-module ghcr.io/nethserver/metrics:1.6.0-dev.2 metrics1
    2. Check that downloaded plugins are gone and Grafana runs from the new image: 
      runagent -m metrics1 podman exec grafana ls -A /var/lib/grafana/plugins
runagent -m metrics1 podman inspect grafana --format '{{.ImageName}}'
runagent -m metrics1 grep -E '^(PREV_)?(METRICS_)?GRAFANA_IMAGE' environment
runagent -m metrics1 podman images --format '{{.Repository}}:{{.Tag}}'
      Expected: empty plugins list; image ghcr.io/nethserver/metrics-grafana:1.6.0-dev.2; only METRICS_GRAFANA_IMAGE set (on core 3.23.0 a harmless PREV_METRICS_GRAFANA_IMAGE=None is also present, fixed by fix(update-module): skip missing previous images ns8-core#1358); no docker.io/grafana/grafana or quay.io images left.
    3. Check that Grafana does not contact grafana.com: 
      runagent -m metrics1 podman logs grafana 2>&1 | grep -c -E 'backgroundinstaller|grafana\.com'
      Expected: 0.
    4. In the Grafana UI: dashboards and data sources work, Drilldown > Logs and Drilldown > Metrics open, and Administration > Plugins does not offer installs.

  14. DavidePrincipi commented on Oct 15, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 15, 2026
    MemberAuthor

    Test results: metrics 1.6.0-dev.1

    Test node: fresh NS8 (core 3.23.0) on Rocky Linux 9, new cluster. metrics1 and loki1 1.5.1 installed by create-cluster; Grafana enabled at /grafana, which downloaded 17 plugins within 40 seconds.

    Step Result
    1. update-module ghcr.io/nethserver/metrics:1.6.0-dev.1 metrics1 ✅ completed; Prometheus, Alertmanager and Grafana active
    2. Plugins in the Grafana volume ✅ none (17 before the update)
    2. Grafana image ✅ ghcr.io/nethserver/metrics-grafana:1.6.0-dev.1
    2. Image variables ⚠️ METRICS_GRAFANA_IMAGE set; PREV_METRICS_GRAFANA_IMAGE=None also present (core issue, see below)
    2. Images left ✅ metrics-grafana, metrics, docker.io/prom/prometheus, docker.io/prom/alertmanager; quay.io/* and docker.io/grafana/grafana removed
    3. backgroundinstaller / grafana.com in the Grafana log ✅ 0
    4. Data sources and dashboards ✅ Loki and Prometheus health OK, 4 dashboards; Alertmanager works through the Grafana proxy (its health check reports "Plugin unavailable", as for any built-in data source without a backend)
    4. Logs and Metrics Drilldown ✅ both 3.0.0, enabled, signed by Grafana Labs, pages answer HTTP 200
    4. Plugin installs ✅ pluginAdminEnabled=false, install API returns 404

    Step 4 was checked through the Grafana API, not in a browser.

    The ⚠️ comes from the core: when an update adds a new image, update-module stores PREV_<NAME>_IMAGE=None and its cleanup step runs podman image rm None, which fails without affecting the update. Fixed by NethServer/ns8-core#1358.

  15. DavidePrincipi commented on Oct 15, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 15, 2026
    MemberAuthor

    Test results: metrics 1.6.0-dev.2

    Same test node, updated from 1.6.0-dev.1 with update-module ghcr.io/nethserver/metrics:1.6.0-dev.2 metrics1. The new release bumps Prometheus to 3.13.4 and Alertmanager from 0.32.1 to 0.34.1.

    • ✅ Update completed; Prometheus, Alertmanager and Grafana active
    • ✅ Images: prom/prometheus:v3.13.4, prom/alertmanager:v0.34.1, metrics-grafana:1.6.0-dev.2; the previous ones were removed by the core
    • ✅ Alertmanager 0.34.1 is ready with the same configuration, and keeps its data: a silence created before the update is still active
    • ✅ Prometheus sends alerts to Alertmanager; no warnings or errors in the Alertmanager log
    • ✅ Test alerts are routed to the default receiver; a critical alert mutes its warning twin (inhibit rule) and the silence mutes its alert

    Alertmanager release notes 0.33–0.34 have no changes affecting the module: the removed auto-gomaxprocs flag and the renamed reason values of alertmanager_notifications_failed_total are not used, and 0.34.1 fixes inhibition bugs. Mail notifications were not tested (no mail relay on the test cluster).

  16. DavidePrincipi
    removed their assignment
    on Oct 15, 2026
  17. DavidePrincipi
    added
    verifiedAll test cases were verified successfully
    on Oct 15, 2026
  18. github-actions
    removed
    testingPackages are available from testing repositories
    on Oct 15, 2026
  19. nethbot
    moved this from Testing to Verified in NethServeron Oct 15, 2026

  20. DavidePrincipi commented on Oct 15, 2026

    @DavidePrincipi
    DavidePrincipi
    on Oct 15, 2026
    MemberAuthor

    Released

  21. DavidePrincipi
    closed this as completedon Oct 15, 2026
  22. github-project-automation
    moved this from Verified to Done in NethServeron Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    verifiedAll test cases were verified successfully

    Type

    Feature

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions