Skip to content

OMD with victoriametrics db shows errors in grafana histou panel #240

Description

@jstark1

We use OMD version 5.71.20261003-labs-edition with Naemon Core and VictoriaMetrics DB. Nagflux collects the data and Histou generates the scripted Grafana dashboard.

In recent OMD versions, we discovered that this dashboard is no longer working. After some investigation, we discovered that the Grafana client is trying to connect to the old InfluxDB, which is no longer running. The generated Histou data looks good and instructs Grafana to connect to the Victoriametrics data source, but Grafana still connects to the old Nagflux (InfluxDB).
The workaround was to set the Victorialogs data source as the Grafana default data source in etc/grafana/provisioning/datasources/victoriametrics.yml by setting "isDefault: true".

I suggest setting it in

.

Activity

  1. sni commented on Oct 5, 2026

    @sni
    Contributor

    there can only be one default datasource, so i'd assume this will very likely break something.

  2. jstark1 commented on Oct 6, 2026

    @jstark1
    Author

    Yes, there can only be one default data source in the provisioned datasources file.

    here a some screenshots:
    nagflux (influxdb) as default datasource
    Image

    no configured default datasource
    Image

    victoriametrics as default datasource
    Image

    i deleted the old datasource with:
    etc/grafana/provisioning/datasources/delete_old_influx_nagflux.yml

    apiVersion: 1
    
    deleteDatasources:
      - name:  nagflux
        orgId: 1
    

    etc/grafana/provisioning/datasources/delete_old_influx_internal.yml

    apiVersion: 1
    
    deleteDatasources:
      - name:  _internal
        orgId: 1
    

    etc/grafana/provisioning/datasources/delete_old_influx_sakuli.yml

    apiVersion: 1
    
    deleteDatasources:
      - name:  sakuli
        orgId: 1
    

    and set isDefault: true in etc/grafana/provisioning/datasources/victoriametrics.yml

    omd restart grafana
    grafanactl datasource
      name: pnp          type: sni-pnp-datasource database:        access: proxy    url: https://127.0.0.1:5000/test/pnp4nagios
      name: thruk        type: sni-thruk-datasource database:        access: proxy    url: https://127.0.0.1:5000/test/thruk
    * name: victoriametrics type: prometheus database:        access: proxy    url: http://127.0.0.1:8431
    

    so everything looks good from my side.

    In OMD the influxdb datasource i the only one that sets "isDefault: true".
    find /omd/versions/default/skel/ -name "datasource.yml" -exec grep "isDefault" {} +
    /omd/versions/default/skel/etc/victoriametrics/grafana_datasources.yml: isDefault: false
    /omd/versions/default/skel/etc/influxdb/grafana_datasources.yml: isDefault: true
    /omd/versions/default/skel/etc/influxdb/grafana_datasources.yml: isDefault: false
    /omd/versions/default/skel/etc/influxdb/grafana_datasources.yml: isDefault: false
    /omd/versions/default/skel/etc/thruk/grafana_datasource.yml: isDefault: false
    /omd/versions/default/skel/etc/promxy/grafana_datasource.yml: isDefault: false
    /omd/versions/default/skel/etc/prometheus/grafana_datasources.yml: isDefault: false
    /omd/versions/default/skel/etc/loki/grafana_datasource.yml: isDefault: false
    /omd/versions/default/skel/etc/loki/grafana_datasource.yml: isDefault: false
    /omd/versions/default/skel/etc/pnp4nagios/grafana_datasource.yml: isDefault: false

    I dont now i it useful to invest in a "default datasource" trigger in omd.

    We are confident that this issue is a corner case caused by our migration from InfluxDB to Victoriametrics, which did not change the default data source.
    I assume that Grafana 13 uses the default data source for some tasks in scripted dashboards. However, as discussed , we must replace this feature anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions