Fix Telegraf options removed in 1.40 on Linux (fieldpass -> fieldinclude, pid_tag -> tag_with) - #1764
Open
suyadav1 wants to merge 1 commit into
Open
Fix Telegraf options removed in 1.40 on Linux (fieldpass -> fieldinclude, pid_tag -> tag_with)#1764suyadav1 wants to merge 1 commit into
suyadav1 wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
The linux agent ships telegraf 1.39.x and logs deprecation warnings for options that telegraf removes in 1.40.0. Telegraf treats an unrecognized option as a fatal config-load error (not a warning), and an empty value such as `fieldpass = []` fails identically, so on 1.40 every linux ama-logs telegraf process would refuse to start -- including on clusters using the default ConfigMap. Migrate the linux configs to the replacement option names: fieldpass -> fieldinclude fielddrop -> fieldexclude pid_tag -> tag_with (inputs.procstat, also removed in 1.40) Covers the daemonset, replicaset, prometheus sidecar and the separate ama-logs process-metrics telegraf instance, plus the prometheus stanzas generated at runtime by tomlparser-prom-customconfig.rb (linux branch) and tomlparser-osm-config.rb. Windows is intentionally left on the legacy option names: it is pinned to telegraf 1.24.2, which predates fieldinclude/fieldexclude/tag_with (added in 1.29.0). tomlparser-prom-customconfig.rb is shared by both platforms, so it now selects the option names via is_windows?, mirroring the existing timeout/response_timeout branch. The customer-facing ConfigMap keys (prometheus_data_collection_settings fieldpass/fielddrop) and the TELEMETRY_* env var names are unchanged; the new names are only used when rendering telegraf TOML. Also drop ignore_protocol_stats from inputs.net. Protocol stat collection was removed in telegraf 1.37.0 and the option is a no-op, so current behavior already matches the previous `true` setting. The startup validation used `--input-filter file -test`, which only loads the dummy file input and so returned success for configs that are fatally invalid, promoting a broken config that then killed telegraf silently at the real (backgrounded, unchecked) start. Replace it with `telegraf config check`, which validates every configured plugin without gathering, so it stays fast and performs no network scrapes. Verified against the telegraf 1.39.1 binary in a live cluster: daemonset config goes from 6 deprecation warnings to 0, process-metrics from 9 to 0, replicaset/sidecar/generated stanzas remain at 0, all with config check RC=0, and inputs.net emits the identical field set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
suyadav1
force-pushed
the
fix/telegraf-fieldinclude-migration
branch
from
August 17, 2026 22:33
a688b2a to
2cdc5af
Compare
Contributor
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Linux agent ships Telegraf 1.39.x, which logs deprecation warnings for options Telegraf removes in 1.40.0:
These read as cosmetic, but they are not. Telegraf treats an unrecognized option as a fatal config-load error, not a warning:
The process exits with RC=1. An empty value (
fieldpass = []) fails identically, so this is not limited to clusters that customizeprometheus_data_collection_settings— a default ConfigMap hits it too. On a 1.40 bump, every Linuxama-logsTelegraf process would refuse to start, dropping InsightsMetrics and breaking the disk-usage MDM alert path (container.azm.ms/disk → used_percent → MdmMetricsGenerator.rb).Fix
Migrate the Linux configs to the replacement option names:
fieldpassfieldincludefielddropfieldexcludepid_tag = truetag_with = ["pid"]Covers the daemonset, replicaset, Prometheus sidecar, and the separate
ama-logsprocess-metrics Telegraf instance, plus the Prometheus stanzas generated at runtime bytomlparser-prom-customconfig.rbandtomlparser-osm-config.rb.pid_tag(inputs.procstat) is not in the warning list above because that config runs as a separate, unguarded Telegraf process whose output isn't in the pod log tail — it is removed in 1.40 all the same, and was caught during validation.Also drops
ignore_protocol_statsfrominputs.net. Protocol-stat collection was removed in 1.37.0 and the option has been a no-op since, so behavior already matches the previoustrue. (This one is scheduled for removal in 1.45, not 1.40.)Windows is deliberately unchanged
Windows is pinned to Telegraf 1.24.2 (
kubernetes/windows/setup.ps1), which predatesfieldinclude/fieldexclude/tag_with— all added in 1.29.0. Renaming there would break it immediately. The Windows configs keep the legacy names, and the comment atsetup.ps1:44now documents the split so the next upgrader doesn't "fix" the inconsistency.build/common/installer/scripts/tomlparser-prom-customconfig.rbis the only file shared by both platforms, so it selects option names viais_windows?, mirroring the existingtimeout/response_timeoutbranch directly above it.Startup validation gate
The gate in
kubernetes/linux/main.shwas:--input-filter filerestricts loading to the dummyfileinput, so it returns RC=0 for a config that is fatally invalid — the broken config gets promoted, and the real (backgrounded, unchecked) Telegraf start then dies silently. I verified this against the 1.39.1 binary: the old gate passed a config that the agent could not actually load.Replaced with
telegraf config check, which validates every configured plugin and performs no gathering. Note that simply dropping--input-filter filefrom-testwould have been a regression —-testreally does scrape kubelet/cadvisor, so it could fail on unready endpoints and silently discard customer Prometheus config.Validation
Run against the real Telegraf 1.39.1 binary in a live AKS cluster, using each config's fully substituted runtime form:
config checktelegraf.conf(daemonset)telegraf-ama-logs-process-metrics.conftelegraf-rs.conf(replicaset)telegraf-prom-side-car.confAlso confirmed:
inputs.netemits an identical field set before/after the rename — no metric drift.telegraf config migrateindependently produces the same mapping for all five affected plugin types (disk,diskio,net,prometheus,procstat). Its output wasn't taken wholesale because it reorders and reformats the entire file; the renames here are surgical.[inputs.disk.tagdrop],[inputs.procstat.tags]) — otherwise they'd bind to the nested table, which is what the existinginfluxdata/telegraf#5615"ORDER matters" comment guards against.is_windows?branch verified functionally: emitsfieldinclude/fieldexcludeon Linux,fieldpass/fielddropon Windows.ruby -cclean on both parsers;bash -nclean onmain.sh.Not changed
fieldpass/fielddropunderprometheus_data_collection_settingsincontainer-azm-ms-agentconfig.yamlare a public API and stay as-is. The new names are used only when rendering Telegraf TOML.TELEMETRY_*_FIELDPASS_LENGTHenv var names.