Skip to content

Fix Telegraf options removed in 1.40 on Linux (fieldpass -> fieldinclude, pid_tag -> tag_with) - #1764

Open
suyadav1 wants to merge 1 commit into
ci_prodfrom
fix/telegraf-fieldinclude-migration
Open

Fix Telegraf options removed in 1.40 on Linux (fieldpass -> fieldinclude, pid_tag -> tag_with)#1764
suyadav1 wants to merge 1 commit into
ci_prodfrom
fix/telegraf-fieldinclude-migration

Conversation

@suyadav1

Copy link
Copy Markdown
Contributor

Problem

The Linux agent ships Telegraf 1.39.x, which logs deprecation warnings for options Telegraf removes in 1.40.0:

W! DeprecationWarning: Option "fieldpass" of plugin "inputs.disk" deprecated since version 1.29.0 and will be removed in 1.40.0: use 'fieldinclude' instead
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.diskio" ...
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.net" ...
W! DeprecationWarning: Option "ignore_protocol_stats" of plugin "inputs.net" deprecated since version 1.37.0 ...
W! DeprecationWarning: Option "fieldpass" of plugin "inputs.prometheus" ... (x2)

These read as cosmetic, but they are not. Telegraf treats an unrecognized option as a fatal config-load error, not a warning:

E! loading config file failed: plugin inputs.disk: line 451:
   configuration specified the fields ["fieldpass"], but they were not used

The process exits with RC=1. An empty value (fieldpass = []) fails identically, so this is not limited to clusters that customize prometheus_data_collection_settings — a default ConfigMap hits it too. On a 1.40 bump, every Linux ama-logs Telegraf process would refuse to start, dropping InsightsMetrics and breaking the disk-usage MDM alert path (container.azm.ms/disk → used_percent → MdmMetricsGenerator.rb).

Fix

Migrate the Linux configs to the replacement option names:

Removed in 1.40 Replacement
fieldpass fieldinclude
fielddrop fieldexclude
pid_tag = true tag_with = ["pid"]

Covers the daemonset, replicaset, Prometheus sidecar, and the separate ama-logs process-metrics Telegraf instance, plus the Prometheus stanzas generated at runtime by tomlparser-prom-customconfig.rb and tomlparser-osm-config.rb.

pid_tag (inputs.procstat) is not in the warning list above because that config runs as a separate, unguarded Telegraf process whose output isn't in the pod log tail — it is removed in 1.40 all the same, and was caught during validation.

Also drops ignore_protocol_stats from inputs.net. Protocol-stat collection was removed in 1.37.0 and the option has been a no-op since, so behavior already matches the previous true. (This one is scheduled for removal in 1.45, not 1.40.)

Windows is deliberately unchanged

Windows is pinned to Telegraf 1.24.2 (kubernetes/windows/setup.ps1), which predates fieldinclude/fieldexclude/tag_with — all added in 1.29.0. Renaming there would break it immediately. The Windows configs keep the legacy names, and the comment at setup.ps1:44 now documents the split so the next upgrader doesn't "fix" the inconsistency.

build/common/installer/scripts/tomlparser-prom-customconfig.rb is the only file shared by both platforms, so it selects option names via is_windows?, mirroring the existing timeout / response_timeout branch directly above it.

Startup validation gate

The gate in kubernetes/linux/main.sh was:

/opt/telegraf --non-strict-env-handling --config <conf> --input-filter file -test

--input-filter file restricts loading to the dummy file input, so it returns RC=0 for a config that is fatally invalid — the broken config gets promoted, and the real (backgrounded, unchecked) Telegraf start then dies silently. I verified this against the 1.39.1 binary: the old gate passed a config that the agent could not actually load.

Replaced with telegraf config check, which validates every configured plugin and performs no gathering. Note that simply dropping --input-filter file from -test would have been a regression — -test really does scrape kubelet/cadvisor, so it could fail on unready endpoints and silently discard customer Prometheus config.

--non-strict-env-handling must come after the config check subcommand; in the global position it is silently ignored.

Validation

Run against the real Telegraf 1.39.1 binary in a live AKS cluster, using each config's fully substituted runtime form:

Config Deprecation warnings config check
telegraf.conf (daemonset) 6 → 0 RC=0
telegraf-ama-logs-process-metrics.conf 9 → 0 RC=0
telegraf-rs.conf (replicaset) 0 → 0 RC=0
telegraf-prom-side-car.conf 0 → 0 RC=0
Generated stanzas (namespaced prom + OSM) 0 → 0 RC=0

Also confirmed:

  • inputs.net emits an identical field set before/after the rename — no metric drift.
  • Upstream's own telegraf config migrate independently produces the same mapping for all five affected plugin types (disk, diskio, net, prometheus, procstat). Its output wasn't taken wholesale because it reorders and reformats the entire file; the renames here are surgical.
  • Key ordering is preserved so plugin-level keys stay above nested tables ([inputs.disk.tagdrop], [inputs.procstat.tags]) — otherwise they'd bind to the nested table, which is what the existing influxdata/telegraf#5615 "ORDER matters" comment guards against.
  • is_windows? branch verified functionally: emits fieldinclude/fieldexclude on Linux, fieldpass/fielddrop on Windows.
  • ruby -c clean on both parsers; bash -n clean on main.sh.

Not changed

  • Customer-facing ConfigMap keysfieldpass/fielddrop under prometheus_data_collection_settings in container-azm-ms-agentconfig.yaml are a public API and stay as-is. The new names are used only when rendering Telegraf TOML.
  • TELEMETRY_*_FIELDPASS_LENGTH env var names.
  • Windows configs (see above).
  • The 1.40 upgrade itself. This makes the codebase 1.40-ready; the version bump is a separate change.

@suyadav1
suyadav1 requested a review from a team as a code owner August 17, 2026 22:31
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

The linux agent ships telegraf 1.39.x and logs deprecation warnings for
options that telegraf removes in 1.40.0. Telegraf treats an unrecognized
option as a fatal config-load error (not a warning), and an empty value
such as `fieldpass = []` fails identically, so on 1.40 every linux
ama-logs telegraf process would refuse to start -- including on clusters
using the default ConfigMap.

Migrate the linux configs to the replacement option names:
  fieldpass  -> fieldinclude
  fielddrop  -> fieldexclude
  pid_tag    -> tag_with  (inputs.procstat, also removed in 1.40)

Covers the daemonset, replicaset, prometheus sidecar and the separate
ama-logs process-metrics telegraf instance, plus the prometheus stanzas
generated at runtime by tomlparser-prom-customconfig.rb (linux branch)
and tomlparser-osm-config.rb.

Windows is intentionally left on the legacy option names: it is pinned to
telegraf 1.24.2, which predates fieldinclude/fieldexclude/tag_with (added
in 1.29.0). tomlparser-prom-customconfig.rb is shared by both platforms,
so it now selects the option names via is_windows?, mirroring the
existing timeout/response_timeout branch.

The customer-facing ConfigMap keys (prometheus_data_collection_settings
fieldpass/fielddrop) and the TELEMETRY_* env var names are unchanged;
the new names are only used when rendering telegraf TOML.

Also drop ignore_protocol_stats from inputs.net. Protocol stat collection
was removed in telegraf 1.37.0 and the option is a no-op, so current
behavior already matches the previous `true` setting.

The startup validation used `--input-filter file -test`, which only loads
the dummy file input and so returned success for configs that are fatally
invalid, promoting a broken config that then killed telegraf silently at
the real (backgrounded, unchecked) start. Replace it with
`telegraf config check`, which validates every configured plugin without
gathering, so it stays fast and performs no network scrapes.

Verified against the telegraf 1.39.1 binary in a live cluster: daemonset
config goes from 6 deprecation warnings to 0, process-metrics from 9 to 0,
replicaset/sidecar/generated stanzas remain at 0, all with config check
RC=0, and inputs.net emits the identical field set.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@suyadav1
suyadav1 force-pushed the fix/telegraf-fieldinclude-migration branch from a688b2a to 2cdc5af Compare August 17, 2026 22:33
@suyadav1

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant