Align ELK role TLS paths and variables to deployer standard

CLAUDE.md

@@ -0,0 +1,132 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Collection Identity
+
+Ansible collection `itential.monitoring` (v1.1.0, namespace: `itential`). Requires Ansible >= 2.9.10.
+
+## Commands
+
+### Run playbooks
+
+```bash
+# Full stacks
+ansible-playbook itential.monitoring.elk -i <inventory>
+ansible-playbook itential.monitoring.site -i <inventory>        # prometheus + grafana + exporters
+
+# Individual components
+ansible-playbook itential.monitoring.loki -i <inventory>
+ansible-playbook itential.monitoring.alloy -i <inventory>
+ansible-playbook itential.monitoring.prometheus -i <inventory>
+ansible-playbook itential.monitoring.grafana -i <inventory>
+ansible-playbook itential.monitoring.prometheus_exporters -i <inventory>
+ansible-playbook itential.monitoring.elasticsearch -i <inventory>
+ansible-playbook itential.monitoring.logstash -i <inventory>
+ansible-playbook itential.monitoring.kibana -i <inventory>
+ansible-playbook itential.monitoring.filebeat -i <inventory>
+```
+
+### Selective execution (tags)
+
+Each role tags its blocks as `<role>_install` and `<role>_configure`. The `always` block (service start + health assert) always runs.
+
+```bash
+ansible-playbook itential.monitoring.loki -i <inventory> --tags loki_configure
+ansible-playbook itential.monitoring.alloy -i <inventory> --tags alloy_configure
+ansible-playbook itential.monitoring.elasticsearch -i <inventory> --tags elasticsearch_configure
+```
+
+### Lint
+
+```bash
+ansible-lint
+```
+
+Config in `.ansible-lint`. The following are warnings (not errors): `yaml[line-length]`, `var-naming[no-role-prefix]`, `meta-runtime[unsupported-version]`, `run-once[task]`.
+
+## Prerequisites
+
+**Prometheus stack** — `prometheus.prometheus` is not installed as a collection dependency and must be added manually:
+
+```bash
+ansible-galaxy collection install prometheus.prometheus  # requires >= 0.22.0
+```
+
+**macOS** — GNU tar is required for Prometheus-related playbooks:
+
+```bash
+brew install gnu-tar
+export PATH="/opt/homebrew/opt/gnu-tar/libexec/gnubin:$PATH"
+export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES   # prevents fork() crash
+```
+
+## Architecture
+
+Three independent monitoring stacks — all optional, deployable together or separately.
+
+### Role structure pattern
+
+Every role uses the same three-block pattern in `tasks/main.yml`:
+
+1. **`<role>_install`** — package/binary install, user/group creation, firewalld rules
+2. **`<role>_configure`** — template deployment; notifies the restart handler
+3. **`always`** — flush handlers → start/enable service → assert `ActiveState == active`
+
+Each role has a single restart handler. Variable defaults live in `defaults/main.yml`; the only exception is `roles/grafana/vars/main.yml`, which holds the internal Prometheus port constant.
+
+---
+
+### Prometheus / Grafana (metrics)
+
+`itential.monitoring.prometheus` is a thin wrapper around the community `prometheus.prometheus.prometheus` role. Its only additions:
+- Generates `{{ prometheus_config_dir }}/scrape_configs/itential.yml` from inventory using `templates/scrape_configs.j2`
+- The scrape template iterates `platform`, `gateway`, `mongodb`, `vault`, and all groups matching `^redis_.*`
+- Per-host port overrides use `<exporter>_web_listen_address` hostvars; the default port from `defaults/main.yml` is used if the hostvar is absent
+
+`itential.monitoring.grafana` is **RHEL/CentOS only** — tasks use `yum_repository`/`dnf` with no `ansible_os_family` guards. Bundled dashboard JSON files live in `roles/grafana/files/definitions/`; adding a file there is enough to include a new dashboard (no task change needed). The Prometheus datasource is provisioned from `groups.prometheus | first`.
+
+`prometheus_exporters.yml` wraps four community exporter roles and adds firewalld rules and — for MongoDB — injects `MONGODB_USER`/`MONGODB_PASSWORD` env vars into the systemd service unit. Redis exporter requires `redis_prometheus_user_enabled: true` on redis hosts. MongoDB exporter requires `mongodb_exporter_global_conn_pool: true` when replication is enabled (prevents file descriptor exhaustion).
+
+`site.yml` chains: exporters → prometheus → grafana.
+
+---
+
+### ELK Stack (centralized logs)
+
+`elk.yml` chains: elasticsearch → logstash → kibana → filebeat.
+
+**TLS is enabled by default** across all four ELK components (`*_tls_enabled: true`). Certificates must be pre-placed on hosts at `/etc/<role>/certs/` before any play runs — no role distributes certs.
+
+**Elasticsearch** defaults to `discovery.type: single-node`. Setting both `elasticsearch_discovery_seed_hosts` and `elasticsearch_cluster_initial_master_nodes` switches to multi-node. JVM heap is deployed as a drop-in to `/etc/elasticsearch/jvm.options.d/itential-heap.options` (not overwriting the main `jvm.options`).
+
+**Logstash** uses a keystore to store `ELASTIC_PASSWORD` at rest. Both `logstash_keystore_password` and `logstash_elastic_password` must be set via Ansible Vault — empty defaults will cause keystore creation to fail. The pipeline (`templates/pipeline.conf.j2`) routes events by the `app` field set in Filebeat input templates; automation failures are dual-written to `itential-failures-<date>`. The keystore task detects Logstash's lowercase key-name output in stdout for idempotency (`changed_when`).
+
+**Filebeat** deploys a drop-in `inputs.d/` directory (`reload.enabled: true`). `main.yml` only deploys `input_default.yml` (syslog). All app-specific inputs are deployed via separate plays in `playbooks/filebeat.yml` using `import_role: tasks_from: filebeat_<app>` — this means app inputs are not deployed when running the role directly; use the `filebeat.yml` playbook.
+
+---
+
+### Loki / Alloy (lightweight log aggregation)
+
+`loki.yml` installs Loki on the `loki` group and — in the same run — calls `tasks_from: loki_datasource` on the `grafana` group to wire the datasource. No separate Grafana playbook step is needed.
+
+**Loki** is installed as a binary from GitHub releases (not via package manager). The binary is idempotent: if `loki --version` already matches `loki_version`, the download is skipped. Single-node filesystem storage only — `auth_enabled: false`. Changing `loki_data_dir` post-deployment requires manually migrating existing data.
+
+**Alloy** replaces Promtail — the install block stops and disables `promtail` (`failed_when: false` so it does not fail if Promtail is absent). Installed via the Grafana RPM repository. The config template (`config.alloy.j2`) is validated with `alloy fmt %s` on every deploy; syntactically invalid HCL will fail the task before writing.
+
+`playbooks/group_vars/` pre-configures `alloy_log_paths` and `alloy_extra_groups` for all standard Itential host groups. The only variable required in inventory is `alloy_loki_url` under `all.vars`. User-defined inventory group_vars take precedence over the collection's playbook group_vars.
+
+`alloy_loki_url` must use the private/VPC-internal IP of the Loki host — EC2 instances cannot route to their own public IP. If this is left as the default empty string, the role deploys a broken config without failing.
+
+`alloy_extra_groups` adds the `alloy` user to OS groups so it can read files not world-readable (e.g. `mongod.log` is `0640 mongod:mongod`). If omitted for a host group that requires it, Alloy silently fails to tail those files.
+
+---
+
+## Secrets (Ansible Vault required in production)
+
+| Variable | Role |
+|---|---|
+| `logstash_keystore_password` | logstash |
+| `logstash_elastic_password` | logstash |
+| `mongodb_exporter_admin_password` | prometheus_exporters |
+| `redis_exporter_password` | prometheus_exporters |

docs/itential_elk_guide.md

@@ -321,8 +321,7 @@ Before running the ELK playbooks:
 
 - Ansible 2.12 or later installed on the control node
 - The `itential.monitoring` collection installed
-- Target hosts running RHEL/CentOS 8+ or Debian/Ubuntu (Filebeat, Logstash,
-  Elasticsearch, Kibana all support both OS families)
+- Target hosts running RHEL/CentOS 8+
 - Internet access to `artifacts.elastic.co` on target hosts, or Elastic packages
   mirrored in a local Nexus repository for air-gapped environments
 - TLS certificates for each component generated and distributed to the target hosts
@@ -334,13 +333,14 @@ Before running the ELK playbooks:
 
 | Host Group | Cert path variables |
 |---|---|
-| `elasticsearch` | `elasticsearch_tls_cert`, `elasticsearch_tls_key`, `elasticsearch_tls_ca_cert` |
-| `logstash` | `logstash_tls_cert`, `logstash_tls_key`, `logstash_tls_ca_cert` |
-| `kibana` | `kibana_tls_cert`, `kibana_tls_key`, `kibana_tls_ca_cert` |
-| All Filebeat hosts | `filebeat_tls_cert`, `filebeat_tls_key`, `filebeat_tls_ca_cert` |
+| `elasticsearch` | `elasticsearch_pki_base_dir`, `elasticsearch_tls_cert_file`, `elasticsearch_tls_key_file`, `elasticsearch_tls_ca_file` |
+| `logstash` | `logstash_pki_base_dir`, `logstash_tls_cert_file`, `logstash_tls_key_file`, `logstash_tls_ca_file` |
+| `kibana` | `kibana_pki_base_dir`, `kibana_tls_cert_file`, `kibana_tls_key_file`, `kibana_tls_ca_file` |
+| All Filebeat hosts | `filebeat_pki_base_dir`, `filebeat_tls_cert_file`, `filebeat_tls_key_file`, `filebeat_tls_ca_file` |
 
-All certificate path variables have defaults under `/etc/<component>/certs/`. Certificates
-should be issued from the same internal CA used for the rest of the Itential deployment.
+Certificate directories follow the `/etc/pki/<component>` standard. Private keys are placed
+in a `private/` subdirectory under the base PKI directory. Certificates should be issued
+from the same internal CA used for the rest of the Itential deployment.
 
 ---
 
@@ -411,10 +411,10 @@ same internal CA.
 
 For a minimal deployment, you need:
 
-- **Elasticsearch host:** `ca.crt`, `elasticsearch.crt`, `elasticsearch.key` under `/etc/elasticsearch/certs/`
-- **Logstash host:** `ca.crt`, `logstash.crt`, `logstash.key` under `/etc/logstash/certs/`
-- **Kibana host:** `ca.crt`, `kibana.crt`, `kibana.key` under `/etc/kibana/certs/`
-- **All Filebeat hosts:** `ca.crt`, `filebeat.crt`, `filebeat.key` under `/etc/filebeat/certs/`
+- **Elasticsearch host:** `ca.crt` and `<hostname>.crt` under `/etc/pki/elasticsearch/`; `<hostname>.key` under `/etc/pki/elasticsearch/private/`
+- **Logstash host:** `ca.crt` and `<hostname>.crt` under `/etc/pki/logstash/`; `<hostname>.key` under `/etc/pki/logstash/private/`
+- **Kibana host:** `ca.crt` and `<hostname>.crt` under `/etc/pki/kibana/`; `<hostname>.key` under `/etc/pki/kibana/private/`
+- **All Filebeat hosts:** `ca.crt` and `<hostname>.crt` under `/etc/pki/filebeat/`; `<hostname>.key` under `/etc/pki/filebeat/private/`
 
 ### Step 3 — Deploy the Full ELK Stack
 

docs/itential_elk_lld.md

@@ -351,8 +351,8 @@ input {
   beats {
     port => 5044
     ssl  => true
-    ssl_certificate => "/etc/logstash/certs/logstash.crt"
-    ssl_key         => "/etc/logstash/certs/logstash.key"
+    ssl_certificate => "/etc/pki/logstash/<hostname>.crt"
+    ssl_key         => "/etc/pki/logstash/private/<hostname>.key"
   }
 }
 
@@ -451,7 +451,7 @@ output {
       index    => "itential-failures-%{+yyyy.MM.dd}"
       user     => "logstash_writer"
       password => "${ELASTIC_PASSWORD}"
-      ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
+      ssl_certificate_authorities => ["/etc/pki/logstash/ca.crt"]
     }
   }
 
@@ -461,7 +461,7 @@ output {
     index    => "itential-logs-%{app}-%{+yyyy.MM.dd}"
     user     => "logstash_writer"
     password => "${ELASTIC_PASSWORD}"
-    ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
+    ssl_certificate_authorities => ["/etc/pki/logstash/ca.crt"]
   }
 }
 ```

roles/elasticsearch/CLAUDE.md

@@ -22,7 +22,7 @@ Installs Elasticsearch from the official Elastic package repository and deploys
 Certificates must be pre-placed on the host before the play runs. The role does not distribute certs.
 
 ## OS support
-Supports both RHEL/CentOS (`rpm_key` + `yum_repository`) and Debian/Ubuntu (`apt_key` + `apt_repository`). All package tasks are guarded with `when: ansible_os_family == "RedHat/Debian"`.
+RHEL/CentOS only. Package install uses `rpm_key` + `yum_repository`.
 
 ## User and role provisioning
 `tasks/create-users.yml` creates the `logstash_writer` role and user, the `grafana_reader` role, and applies the `itential-logs-policy` ILM policy via the Elasticsearch REST API. It runs after the service is confirmed active and is tagged `elasticsearch_users`. Both `elasticsearch_elastic_password` and `elasticsearch_logstash_writer_password` must be set (use Ansible Vault).

roles/elasticsearch/README.md

@@ -4,7 +4,7 @@ Installs and configures Elasticsearch for the Itential monitoring stack.
 
 ## Requirements
 
-- RHEL/CentOS 8+ or Debian/Ubuntu
+- RHEL/CentOS 8+
 - Systemd
 - Internet access to the Elastic package repository (or a local mirror)
 
@@ -42,11 +42,11 @@ These have empty string defaults and must be set in inventory using Ansible Vaul
 |---|---|---|
 | `elasticsearch_tls_enabled` | `true` | Enable xpack security and TLS for HTTP and transport |
 | `elasticsearch_tls_copy_certs` | `true` | Copy certificates from the control node to the target host |
-| `elasticsearch_tls_src_dir` | `""` | Directory on the control node containing the certificate files |
-| `elasticsearch_tls_dest_dir` | `/etc/elasticsearch/certs` | Directory on the target host where certificates are placed |
-| `elasticsearch_tls_cert_filename` | `{{ inventory_hostname }}.crt` | Certificate filename |
-| `elasticsearch_tls_key_filename` | `{{ inventory_hostname }}.key` | Private key filename |
-| `elasticsearch_tls_ca_cert_filename` | `ca.crt` | CA certificate filename |
+| `elasticsearch_pki_src_dir` | `""` | Directory on the control node containing the certificate files |
+| `elasticsearch_pki_base_dir` | `/etc/pki/elasticsearch` | Base PKI directory on the target host |
+| `elasticsearch_tls_cert_file` | `{{ inventory_hostname }}.crt` | Certificate filename |
+| `elasticsearch_tls_key_file` | `{{ inventory_hostname }}.key` | Private key filename |
+| `elasticsearch_tls_ca_file` | `ca.crt` | CA certificate filename |
 
 ### ILM
 

roles/elasticsearch/defaults/main.yml

@@ -31,17 +31,19 @@ elasticsearch_heap_size: "1g"
 # all three flags must be set consistently to avoid a startup conflict.
 elasticsearch_tls_enabled: true
 elasticsearch_tls_copy_certs: true
-elasticsearch_tls_src_dir: ""
-elasticsearch_tls_dest_dir: /etc/elasticsearch/certs
-elasticsearch_tls_cert_filename: "{{ inventory_hostname }}.crt"
-elasticsearch_tls_cert_src: "{{ elasticsearch_tls_src_dir }}/{{ elasticsearch_tls_cert_filename }}"
-elasticsearch_tls_cert_dest: "{{ elasticsearch_tls_dest_dir }}/{{ elasticsearch_tls_cert_filename }}"
-elasticsearch_tls_key_filename: "{{ inventory_hostname }}.key"
-elasticsearch_tls_key_src: "{{ elasticsearch_tls_src_dir }}/{{ elasticsearch_tls_key_filename }}"
-elasticsearch_tls_key_dest: "{{ elasticsearch_tls_dest_dir }}/{{ elasticsearch_tls_key_filename }}"
-elasticsearch_tls_ca_cert_filename: ca.crt
-elasticsearch_tls_ca_cert_src: "{{ elasticsearch_tls_src_dir }}/{{ elasticsearch_tls_ca_cert_filename }}"
-elasticsearch_tls_ca_cert_dest: "{{ elasticsearch_tls_dest_dir }}/{{ elasticsearch_tls_ca_cert_filename }}"
+elasticsearch_pki_src_dir: ""
+elasticsearch_pki_base_dir: /etc/pki/elasticsearch
+elasticsearch_pki_private_subdir: private
+elasticsearch_pki_private_dir: "{{ elasticsearch_pki_base_dir }}/{{ elasticsearch_pki_private_subdir }}"
+elasticsearch_tls_cert_file: "{{ inventory_hostname }}.crt"
+elasticsearch_tls_cert_src: "{{ elasticsearch_pki_src_dir }}/{{ elasticsearch_tls_cert_file }}"
+elasticsearch_tls_cert_dest: "{{ elasticsearch_pki_base_dir }}/{{ elasticsearch_tls_cert_file }}"
+elasticsearch_tls_key_file: "{{ inventory_hostname }}.key"
+elasticsearch_tls_key_src: "{{ elasticsearch_pki_src_dir }}/{{ elasticsearch_tls_key_file }}"
+elasticsearch_tls_key_dest: "{{ elasticsearch_pki_private_dir }}/{{ elasticsearch_tls_key_file }}"
+elasticsearch_tls_ca_file: ca.crt
+elasticsearch_tls_ca_src: "{{ elasticsearch_pki_src_dir }}/{{ elasticsearch_tls_ca_file }}"
+elasticsearch_tls_ca_dest: "{{ elasticsearch_pki_base_dir }}/{{ elasticsearch_tls_ca_file }}"
 
 # System user/group created by the package
 elasticsearch_user: elasticsearch

roles/elasticsearch/tasks/copy-certs.yml

@@ -7,9 +7,17 @@
     - elasticsearch_certificates
     - elasticsearch_certs
   block:
-    - name: Create Elasticsearch TLS directory
+    - name: Create Elasticsearch PKI base directory
       ansible.builtin.file:
-        path: "{{ elasticsearch_tls_dest_dir }}"
+        path: "{{ elasticsearch_pki_base_dir }}"
+        state: directory
+        owner: "{{ elasticsearch_user }}"
+        group: "{{ elasticsearch_group }}"
+        mode: "0750"
+
+    - name: Create Elasticsearch PKI private directory
+      ansible.builtin.file:
+        path: "{{ elasticsearch_pki_private_dir }}"
         state: directory
         owner: "{{ elasticsearch_user }}"
         group: "{{ elasticsearch_group }}"
@@ -20,8 +28,8 @@
       block:
         - name: Copy the CA certificate file
           ansible.builtin.copy:
-            src: "{{ elasticsearch_tls_ca_cert_src }}"
-            dest: "{{ elasticsearch_tls_ca_cert_dest }}"
+            src: "{{ elasticsearch_tls_ca_src }}"
+            dest: "{{ elasticsearch_tls_ca_dest }}"
             mode: "0644"
             owner: "{{ elasticsearch_user }}"
             group: "{{ elasticsearch_group }}"
@@ -42,34 +50,16 @@
             owner: "{{ elasticsearch_user }}"
             group: "{{ elasticsearch_group }}"
 
-    - name: Add Itential CA to system trust store (RedHat)
+    - name: Add Itential CA to system trust store
       ansible.builtin.copy:
-        src: "{{ elasticsearch_tls_ca_cert_dest }}"
+        src: "{{ elasticsearch_tls_ca_dest }}"
         dest: /etc/pki/ca-trust/source/anchors/itential-ca.crt
         remote_src: true
         owner: root
         group: root
         mode: '0644'
-      when: ansible_os_family == "RedHat"
 
-    - name: Add Itential CA to system trust store (Debian)
-      ansible.builtin.copy:
-        src: "{{ elasticsearch_tls_ca_cert_dest }}"
-        dest: /usr/local/share/ca-certificates/itential-ca.crt
-        remote_src: true
-        owner: root
-        group: root
-        mode: '0644'
-      when: ansible_os_family == "Debian"
-
-    - name: Update system CA trust store (RedHat)
+    - name: Update system CA trust store
       ansible.builtin.command:
         cmd: update-ca-trust extract
       changed_when: false
-      when: ansible_os_family == "RedHat"
-
-    - name: Update system CA trust store (Debian)
-      ansible.builtin.command:
-        cmd: update-ca-certificates
-      changed_when: false
-      when: ansible_os_family == "Debian"

roles/elasticsearch/tasks/main.yml

@@ -20,22 +20,9 @@
         enabled: true
       when: ansible_os_family == "RedHat"
 
-    - name: Add Elasticsearch APT GPG key
-      ansible.builtin.apt_key:
-        url: https://artifacts.elastic.co/GPG-KEY-elasticsearch
-        state: present
-      when: ansible_os_family == "Debian"
-
-    - name: Add Elasticsearch APT repository
-      ansible.builtin.apt_repository:
-        repo: "deb https://artifacts.elastic.co/packages/{{ elasticsearch_version.split('.')[0] }}.x/apt stable main"
-        state: present
-        filename: elasticsearch
-      when: ansible_os_family == "Debian"
-
     - name: Install Elasticsearch package
       ansible.builtin.package:
-        name: "elasticsearch{% if ansible_os_family == 'RedHat' %}-{{ elasticsearch_version }}{% endif %}"
+        name: "elasticsearch-{{ elasticsearch_version }}"
         state: present
 
 - name: Configure Elasticsearch