Skip to content

Add documentation on scaling integration policies - #8208

Open
vishaangelova wants to merge 3 commits into
mainfrom
2166-integration-scaling-doc
Open

vishaangelova wants to merge 3 commits into
mainfrom
2166-integration-scaling-doc

Conversation

@vishaangelova

@vishaangelova vishaangelova commented Sep 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Customers who monitor many hosts running the same service had no documented way to avoid creating one integration policy per host.

This PR adds a new doc, Scale integration policies across many hosts (scale-integration-policies.md), under Fleet and Elastic Agent → Manage integrations. It documents the technique from the Support KB article linked in the issue, explains why a shared host list doesn't work, and covers the secondary consolidation options: sharing an integration policy across agent policies and grouping agents by role.

Closes #2166

Other changes

  • Folds fleet-agent-environment-variables.md into the new page and deletes it. The published URL is preserved through mapped_pages and a redirects.yml entry that maps the old anchor to #reuse-one-integration-policy-with-variables.
  • Adds entry points from related docs.

Verification

The Support KB article referenced in the linked issue is the source of the described technique: the embedded ${env.ORACLE_DB_ADDRESS} pattern in Oracle DSN, the systemd override with Environment= or EnvironmentFile=, and verifying provider values through the diagnostics bundle.

Product claims were checked against the code and existing docs rather than the issue or KB text.

Source checks
  • title: Oracle DSN on the hosts variable, multi: true, default oracle://0.0.0.0:1521/ORCLCDB.localdomain?sysdba=1; password is type: password, secret: true — elastic/integrations, packages/oracle/manifest.yml. The multi-value field is what makes the anti-pattern tempting, which is why the page leads with it, and the secret flag is what backs the guidance to keep credentials in the integration's own fields.
  • Fleet passes ${env.X} through to the agent unchanged. Integration templates are compiled with Handlebars noEscape: true and there is no special handling of $ — elastic/kibana, fleet/server/services/epm/agent/agent.ts.
  • New hosts / Existing hosts tabs — fleet/public/.../steps/step_select_hosts.tsx.
  • Reusable integration policies gated on an Enterprise license — fleet/public/hooks/use_multiple_agent_policies.ts, LICENCE_FOR_MULTIPLE_AGENT_POLICIES = 'enterprise' in fleet/common/constants/package_policy.ts.
  • Actions → View policy — agent policy actions_menu.tsx.
  • elastic-agent inspect --variables — elastic/elastic-agent, internal/pkg/agent/cmd/inspect.go.
  • Provider settings (as opposed to provider variables) can't be managed from Fleet. There's no UI field for them, the API-only agent policy overrides is documented as not for routine use (fleet/server/types/models/agent_policy.ts), and on the agent side the providers controller is built once from the local elastic-agent.yml (internal/pkg/agent/application/application.go, composable.New) with no reload path from the policy (internal/pkg/composable/controller.go). The only channel is editing elastic-agent.yml on each host, which is not a workflow to recommend to Fleet users, so the KB's local-provider technique is left out of the page. Environment variables cover the same per-host need.
  • variables.yaml and computed-config.yaml in the diagnostics bundle — diagnostics hooks in internal/pkg/agent/application/coordinator/coordinator.go; both files are already listed in agent-command-reference.md#elastic-agent-diagnostics-command. The KB says variables.yaml sits in the agent install directory; source shows it's produced only as part of diagnostics, so the page describes it that way.
  • Undefined variable removes the input, | fallback chaining, $$ escaping — consistent with dynamic-input-configuration.md.
  • Policy limits (1000 / 500 agent policies, 10,000 integration policies per agent policy) — consistent with agent-policy.md.

Vale is clean apart from the expected Elastic.MappedPages warning, which is there because the new page takes over the deleted page's legacy URL.

Generative AI disclosure

  1. Did you use a generative AI (GenAI) tool to assist in creating this contribution?
  • Yes
  • No
  1. Tool(s) and model(s) used: Cursor (Claude Opus 5, Claude Fable 5.1)

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Elastic Docs AI PR menu

Check the box to run an AI review for this pull request.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 1 warning found

⚠️ Warnings (1): Fix when the suggestion improves clarity or correctness.
File Line Rule Message
reference/fleet/scale-integration-policies.md 3 Elastic.MappedPages Avoid editing mapped_pages unless you are preserving an existing URL mapping. Talk with your local technical writer before changing this key.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@vishaangelova
vishaangelova force-pushed the 2166-integration-scaling-doc branch from e6dd9b8 to 03d71ec Compare September 22, 2026 14:02
@vishaangelova
vishaangelova force-pushed the 2166-integration-scaling-doc branch from 03d71ec to 6859266 Compare September 22, 2026 14:16
@vishaangelova vishaangelova changed the title [WIP] Add guidance on reducing the number of Fleet integration policies Add guidance on reducing the number of Fleet integration policies Sep 22, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs review summary

Criteria

  • User focus: The new page addresses a real pain point (per-host integration policies) with a concrete worked example (Oracle DSN) and cross-links from agent-policy.md, providers.md, env-provider.md, add-integration-to-policy.md, and the manage-integrations.md table, so readers reach it from the places they'd naturally look.
  • Technical accuracy: Commands, file paths, and anchors referenced (elastic-agent inspect --variables, variables.yaml/computed-config.yaml, fallback syntax ${env.VAR1|env.VAR2|'default'}, proxy env var anchor) all resolve correctly against the current repo content. No cross-repo contradictions surfaced via elastic-docs search on agent/integration policy limits or reusable-policy licensing — the closest published matches (reference/fleet/agent-policy) match the new page's claims.
  • Applicability: applies_to: {stack: ga, serverless: ga} matches the sibling pages it cross-links from. The explicit deferral of "integration policy conditions (9.5)" to a follow-up PR keeps scope disciplined.
  • Maintainability: The mapped_pages/redirects.yml entry for the deleted fleet-agent-environment-variables.md stub correctly redirects to the new page's reuse-one-integration-policy-with-variables anchor, and no other file still references the old path. toc.yml placement is consistent with the surrounding integrations section.
  • Language: No jargon issues beyond the repo's existing {{agent}}/{{fleet}} substitution conventions, which this page uses the same way as its siblings.
  • Style: All pre-fetched Vale findings in the touched files (agent-policy.md, manage-integrations.md, providers.md) land on pre-existing lines untouched by this diff, so none apply to the new content itself. One manual UI-writing deviation found — see inline comment.
  • Issue satisfaction: Satisfied — the page gives readers a way to scale integration policies without one-per-host proliferation, matching the linked issue's ask.

Nits

  • reference/fleet/scale-integration-policies.md has no type: frontmatter field. This is a repo-wide gap across nearly all of reference/fleet/*.md (only the OTel collector pages set it), so it's not specific to this PR, but worth setting if the team starts backfilling this field.

Notes

  • Cross-repo contradiction check ran via the elastic-docs MCP (search_docs) against the policy-limit and license-gating claims; no conflicting published guidance found.

Generated by Docs review agent for #8208 · claude · sonnet · 215.4 AIC · ⊞ 19K

Comment thread reference/fleet/scale-integration-policies.md Outdated
@vishaangelova
vishaangelova marked this pull request as ready for review September 22, 2026 14:50
@vishaangelova
vishaangelova requested review from a team as code owners September 22, 2026 14:50
@vishaangelova vishaangelova changed the title Add guidance on reducing the number of Fleet integration policies Add documentation on scaling integration policies Sep 22, 2026
@vishaangelova
vishaangelova force-pushed the 2166-integration-scaling-doc branch from 88e085c to c8cef9a Compare September 22, 2026 15:13

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs review summary

Criteria

  • User focus: The new page addresses a concrete pain point (one integration policy per host) with a clear problem framing and three actionable strategies (variables, shared policies, grouping by role). Good fit for a Fleet administrator's mental model.
  • Technical accuracy: Scaling numbers (1000/500 agent policies, 10,000 integration policies) are self-consistent with agent-policy.md. One contradiction found and flagged inline on providers.md: it frames the elastic-agent.yml providers block as standalone-only, while the new page states {{agent}} reads that block "whether it's standalone or {{fleet}}-managed" — the PR body backs this with source citations, so providers.md likely needs the update, but an SME should confirm before merging either side further.
  • Applicability: applies_to (stack: ga, serverless: ga) and products (fleet, elastic-agent) frontmatter are correctly scoped. The reusable-integration-policies subscription note appropriately avoids naming a specific tier and links to the canonical subscriptions page instead.
  • Maintainability: Cross-references reuse existing pages (host-proxy-env-vars.md, dynamic-input-configuration.md, agent-policy.md) rather than duplicating their content — good separation of concerns.
  • Language: Active voice, second person, present tense throughout; no word-choice violations found in the diff.
  • Style: Vale's 26 findings for this PR's files all fall outside the changed lines (pre-existing, out of scope); no new style issues in the diff itself.
  • Issue satisfaction: Closes #2166 by adding the missing scaling-guidance page and linking it from the pages where the gap was felt (agent-policy.md, add-integration-to-policy.md, env-provider.md, manage-integrations.md, providers.md).

Nits

  • reference/fleet/scale-integration-policies.md has no type: frontmatter field. A how-to type fits the page's task-oriented, numbered structure — worth adding even though nothing currently consumes the field.

Notes

  • See the inline comment on reference/fleet/providers.md:107 for the one contradiction found: the page's standalone/Fleet-managed split doesn't yet reflect the new page's claim (source-cited in the PR description) that a providers block in elastic-agent.yml also applies to Fleet-managed agents. Direction: review existing (the new content looks like a deliberate, source-verified update), severity: medium (inconsistent information, not a broken outcome for readers who follow the cross-reference to the new page).
  • Contradiction check covered the local repo only (files in this PR's diff plus their existing cross-referenced pages); the live published providers.md page matches the local version, so the cross-repo check via the elastic-docs MCP didn't surface additional conflicts.

Generated by Docs review agent for #8208 · claude · sonnet · 207.6 AIC · ⊞ 19K

Comment thread reference/fleet/providers.md
Provider settings can only be changed in elastic-agent.yml on the host,
which is not a workflow to recommend to Fleet users, and environment
variables already cover the per-host use case. Removing the local
provider row and example also resolves the contradiction with
providers.md flagged in review.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vishaangelova

Copy link
Copy Markdown
Member Author

@bturquet, could you suggest an SME from your team who could review this PR? Thanks!


An integration policy applies to *every* {{agent}} enrolled in the {{agent}} policy that contains it. If you list 200 database hosts, each of the 200 agents receives the full list and tries to connect to all 200 databases, not only the one running on its own host.

Agents that can't reach the other databases report failed connections, and the integration shows as unhealthy. Agents that can reach them collect the same data from every database, so you store 200 copies of every metric.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: while it's true each EA would collect the same metrics, if the destination data streams are TSDS, most of the duplicate events (determined by the dimensions of the metric and the timestamp) will be rejected by Elasticsearch.

The waste of the resources to run 200 EA is real - as well as the resources on ES to reject the events.

It might be acceptable to have 2 EA pulling the same metrics as a poor attempt to implement HA.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ea28e93


| Provider | Example variables | Use it for |
| --- | --- | --- |
| [Env](/reference/fleet/env-provider.md) | `${env.VAR_NAME}` | Values you define per host as environment variables, such as a database address or a log directory. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would mention env vars will need to be deployed by the user via other means (EA cannot deploy env vars), like configuration management tools.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ea28e93

Comment thread reference/fleet/scale-integration-policies.md
- id: elastic-agent
---

# Scale integration policies across many hosts [scale-integration-policies]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the content added is generally valid.

We will need to be careful and address some more advanced concepts like Integrations which can be scaled horizontally. Example at elastic/package-spec#842

@vishaangelova vishaangelova Sep 23, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ea28e93, I added a short note in the introduction that some integrations collect from a shared source and that adding agents increases throughput without duplicate data.

Documenting the scaling model for each integration is tracked in #7761, the follow-up from elastic/package-spec#842. We'll cover that there.

@bturquet

Copy link
Copy Markdown

@criamico @juliaElastic can you help @vishaangelova with the review ? thanks

@juliaElastic

Copy link
Copy Markdown
Contributor

The Fleet related content LGTM. About the Agent / envvar related parts I would defer to Agent team cc @ebeahan

This branch was successfully deployed

1 active deployment
docs-preview — ea28e93e Deployed Sep 23, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Internal]: Fleet Managed Integration Scaling document improvement

4 participants