Add documentation on scaling integration policies - #8208
vishaangelova wants to merge 3 commits into
Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
🔍 Preview links for changed docs |
Elastic Docs Style Checker (Vale)Summary: 1 warning found
|
| File | Line | Rule | Message |
|---|---|---|---|
| reference/fleet/scale-integration-policies.md | 3 | Elastic.MappedPages | Avoid editing mapped_pages unless you are preserving an existing URL mapping. Talk with your local technical writer before changing this key. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
e6dd9b8 to
03d71ec
Compare
03d71ec to
6859266
Compare
There was a problem hiding this comment.
Docs review summary
Criteria
- User focus: The new page addresses a real pain point (per-host integration policies) with a concrete worked example (Oracle DSN) and cross-links from
agent-policy.md,providers.md,env-provider.md,add-integration-to-policy.md, and themanage-integrations.mdtable, so readers reach it from the places they'd naturally look. - Technical accuracy: Commands, file paths, and anchors referenced (
elastic-agent inspect --variables,variables.yaml/computed-config.yaml, fallback syntax${env.VAR1|env.VAR2|'default'}, proxy env var anchor) all resolve correctly against the current repo content. No cross-repo contradictions surfaced viaelastic-docssearch on agent/integration policy limits or reusable-policy licensing — the closest published matches (reference/fleet/agent-policy) match the new page's claims. - Applicability:
applies_to: {stack: ga, serverless: ga}matches the sibling pages it cross-links from. The explicit deferral of "integration policy conditions (9.5)" to a follow-up PR keeps scope disciplined. - Maintainability: The
mapped_pages/redirects.ymlentry for the deletedfleet-agent-environment-variables.mdstub correctly redirects to the new page'sreuse-one-integration-policy-with-variablesanchor, and no other file still references the old path.toc.ymlplacement is consistent with the surrounding integrations section. - Language: No jargon issues beyond the repo's existing
{{agent}}/{{fleet}}substitution conventions, which this page uses the same way as its siblings. - Style: All pre-fetched Vale findings in the touched files (
agent-policy.md,manage-integrations.md,providers.md) land on pre-existing lines untouched by this diff, so none apply to the new content itself. One manual UI-writing deviation found — see inline comment. - Issue satisfaction: Satisfied — the page gives readers a way to scale integration policies without one-per-host proliferation, matching the linked issue's ask.
Nits
reference/fleet/scale-integration-policies.mdhas notype:frontmatter field. This is a repo-wide gap across nearly all ofreference/fleet/*.md(only the OTel collector pages set it), so it's not specific to this PR, but worth setting if the team starts backfilling this field.
Notes
- Cross-repo contradiction check ran via the
elastic-docsMCP (search_docs) against the policy-limit and license-gating claims; no conflicting published guidance found.
Generated by Docs review agent for #8208 · claude · sonnet · 215.4 AIC · ⊞ 19K
88e085c to
c8cef9a
Compare
There was a problem hiding this comment.
Docs review summary
Criteria
- User focus: The new page addresses a concrete pain point (one integration policy per host) with a clear problem framing and three actionable strategies (variables, shared policies, grouping by role). Good fit for a Fleet administrator's mental model.
- Technical accuracy: Scaling numbers (1000/500 agent policies, 10,000 integration policies) are self-consistent with
agent-policy.md. One contradiction found and flagged inline onproviders.md: it frames theelastic-agent.ymlprovidersblock as standalone-only, while the new page states {{agent}} reads that block "whether it's standalone or {{fleet}}-managed" — the PR body backs this with source citations, soproviders.mdlikely needs the update, but an SME should confirm before merging either side further. - Applicability:
applies_to(stack: ga,serverless: ga) andproducts(fleet,elastic-agent) frontmatter are correctly scoped. The reusable-integration-policies subscription note appropriately avoids naming a specific tier and links to the canonical subscriptions page instead. - Maintainability: Cross-references reuse existing pages (
host-proxy-env-vars.md,dynamic-input-configuration.md,agent-policy.md) rather than duplicating their content — good separation of concerns. - Language: Active voice, second person, present tense throughout; no word-choice violations found in the diff.
- Style: Vale's 26 findings for this PR's files all fall outside the changed lines (pre-existing, out of scope); no new style issues in the diff itself.
- Issue satisfaction: Closes #2166 by adding the missing scaling-guidance page and linking it from the pages where the gap was felt (
agent-policy.md,add-integration-to-policy.md,env-provider.md,manage-integrations.md,providers.md).
Nits
reference/fleet/scale-integration-policies.mdhas notype:frontmatter field. Ahow-totype fits the page's task-oriented, numbered structure — worth adding even though nothing currently consumes the field.
Notes
- See the inline comment on
reference/fleet/providers.md:107for the one contradiction found: the page's standalone/Fleet-managed split doesn't yet reflect the new page's claim (source-cited in the PR description) that aprovidersblock inelastic-agent.ymlalso applies to Fleet-managed agents. Direction: review existing (the new content looks like a deliberate, source-verified update), severity: medium (inconsistent information, not a broken outcome for readers who follow the cross-reference to the new page). - Contradiction check covered the local repo only (files in this PR's diff plus their existing cross-referenced pages); the live published
providers.mdpage matches the local version, so the cross-repo check via the elastic-docs MCP didn't surface additional conflicts.
Generated by Docs review agent for #8208 · claude · sonnet · 207.6 AIC · ⊞ 19K
Provider settings can only be changed in elastic-agent.yml on the host, which is not a workflow to recommend to Fleet users, and environment variables already cover the per-host use case. Removing the local provider row and example also resolves the contradiction with providers.md flagged in review. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@bturquet, could you suggest an SME from your team who could review this PR? Thanks! |
|
|
||
| An integration policy applies to *every* {{agent}} enrolled in the {{agent}} policy that contains it. If you list 200 database hosts, each of the 200 agents receives the full list and tries to connect to all 200 databases, not only the one running on its own host. | ||
|
|
||
| Agents that can't reach the other databases report failed connections, and the integration shows as unhealthy. Agents that can reach them collect the same data from every database, so you store 200 copies of every metric. |
There was a problem hiding this comment.
Nit: while it's true each EA would collect the same metrics, if the destination data streams are TSDS, most of the duplicate events (determined by the dimensions of the metric and the timestamp) will be rejected by Elasticsearch.
The waste of the resources to run 200 EA is real - as well as the resources on ES to reject the events.
It might be acceptable to have 2 EA pulling the same metrics as a poor attempt to implement HA.
|
|
||
| | Provider | Example variables | Use it for | | ||
| | --- | --- | --- | | ||
| | [Env](/reference/fleet/env-provider.md) | `${env.VAR_NAME}` | Values you define per host as environment variables, such as a database address or a log directory. | |
There was a problem hiding this comment.
I would mention env vars will need to be deployed by the user via other means (EA cannot deploy env vars), like configuration management tools.
| - id: elastic-agent | ||
| --- | ||
|
|
||
| # Scale integration policies across many hosts [scale-integration-policies] |
There was a problem hiding this comment.
All the content added is generally valid.
We will need to be careful and address some more advanced concepts like Integrations which can be scaled horizontally. Example at elastic/package-spec#842
There was a problem hiding this comment.
In ea28e93, I added a short note in the introduction that some integrations collect from a shared source and that adding agents increases throughput without duplicate data.
Documenting the scaling model for each integration is tracked in #7761, the follow-up from elastic/package-spec#842. We'll cover that there.
|
@criamico @juliaElastic can you help @vishaangelova with the review ? thanks |
|
The Fleet related content LGTM. About the Agent / envvar related parts I would defer to Agent team cc @ebeahan |
Summary
Customers who monitor many hosts running the same service had no documented way to avoid creating one integration policy per host.
This PR adds a new doc, Scale integration policies across many hosts (
scale-integration-policies.md), under Fleet and Elastic Agent → Manage integrations. It documents the technique from the Support KB article linked in the issue, explains why a shared host list doesn't work, and covers the secondary consolidation options: sharing an integration policy across agent policies and grouping agents by role.Closes #2166
Other changes
fleet-agent-environment-variables.mdinto the new page and deletes it. The published URL is preserved throughmapped_pagesand aredirects.ymlentry that maps the old anchor to#reuse-one-integration-policy-with-variables.Verification
The Support KB article referenced in the linked issue is the source of the described technique: the embedded
${env.ORACLE_DB_ADDRESS}pattern in Oracle DSN, the systemd override withEnvironment=orEnvironmentFile=, and verifying provider values through the diagnostics bundle.Product claims were checked against the code and existing docs rather than the issue or KB text.
Source checks
title: Oracle DSNon thehostsvariable,multi: true, defaultoracle://0.0.0.0:1521/ORCLCDB.localdomain?sysdba=1;passwordistype: password,secret: true—elastic/integrations,packages/oracle/manifest.yml. The multi-value field is what makes the anti-pattern tempting, which is why the page leads with it, and the secret flag is what backs the guidance to keep credentials in the integration's own fields.${env.X}through to the agent unchanged. Integration templates are compiled with HandlebarsnoEscape: trueand there is no special handling of$—elastic/kibana,fleet/server/services/epm/agent/agent.ts.fleet/public/.../steps/step_select_hosts.tsx.fleet/public/hooks/use_multiple_agent_policies.ts,LICENCE_FOR_MULTIPLE_AGENT_POLICIES = 'enterprise'infleet/common/constants/package_policy.ts.actions_menu.tsx.elastic-agent inspect --variables—elastic/elastic-agent,internal/pkg/agent/cmd/inspect.go.overridesis documented as not for routine use (fleet/server/types/models/agent_policy.ts), and on the agent side the providers controller is built once from the localelastic-agent.yml(internal/pkg/agent/application/application.go,composable.New) with no reload path from the policy (internal/pkg/composable/controller.go). The only channel is editingelastic-agent.ymlon each host, which is not a workflow to recommend to Fleet users, so the KB's local-provider technique is left out of the page. Environment variables cover the same per-host need.variables.yamlandcomputed-config.yamlin the diagnostics bundle — diagnostics hooks ininternal/pkg/agent/application/coordinator/coordinator.go; both files are already listed inagent-command-reference.md#elastic-agent-diagnostics-command. The KB saysvariables.yamlsits in the agent install directory; source shows it's produced only as part of diagnostics, so the page describes it that way.|fallback chaining,$$escaping — consistent withdynamic-input-configuration.md.agent-policy.md.Vale is clean apart from the expected
Elastic.MappedPageswarning, which is there because the new page takes over the deleted page's legacy URL.Generative AI disclosure