Repository navigation
fix: Security adjustments - #326
Conversation
3a7f190 to
1fe7bb5
Compare
1fe7bb5 to
946d6a1
Compare
| - users | ||
| - k8s |
There was a problem hiding this comment.
Reorder ansible tags and put more relevant at the top
| alert_email: ${{ secrets.ALERT_EMAIL }} | ||
| backup_host_public_key: ${{ secrets.BACKUP_HOST_PUBLIC_KEY }} | ||
| kube_cluster_env: ${{ inputs.environment }} | ||
| kube_api_host: ${{ vars.KUBE_API_HOST }} |
There was a problem hiding this comment.
This variable was mistakenly missed, adding it back
| validate: validateCIDRs, | ||
| valueLabel: 'KUBE_API_ALLOWED_CIDRS', | ||
| initial: process.env.KUBE_API_ALLOWED_CIDRS || "0.0.0.0/0", | ||
| initial: process.env.KUBE_API_ALLOWED_CIDRS || "", |
There was a problem hiding this comment.
Deny everything by default
| # Ports required for Kubernetes cluster node-to-node communication (used for firewall/UFW rules) | ||
| cluster_communication_ports: | ||
| # Kubelet API (required for all nodes) | ||
| - { port: 10250, proto: "tcp" } | ||
| # kube-scheduler (control plane only) | ||
| - { port: 10251, proto: "tcp" } | ||
| # kube-controller-manager (control plane only) | ||
| - { port: 10252, proto: "tcp" } | ||
| # Overlay network (Calico/Weave/Flannel, if used—verify your CNI and usage) | ||
| - { port: 7946, proto: "tcp" } # Used by some CNIs (e.g., Flannel, Weave) | ||
| - { port: 7946, proto: "udp" } # Used by some CNIs (e.g., Flannel, Weave) | ||
| - { port: 8472, proto: "udp" } # VXLAN (Flannel/Calico; verify if needed) | ||
| - { port: 4789, proto: "udp" } # VXLAN (Calico; verify if needed) | ||
| - { port: 179, proto: "tcp" } # BGP (Calico) | ||
| - { port: 179, proto: "udp" } # BGP (Calico) |
There was a problem hiding this comment.
Move allowed ports definition to configuration file from firewall rules
There was a problem hiding this comment.
In this file kubelet is configured to private IP address
| shell: | | ||
| sudo kubeadm init \ | ||
| --apiserver-advertise-address={{ kube_api_address | default(ansible_default_ipv4.address) }} \ | ||
| --apiserver-advertise-address={{ kube_api_host | default(ansible_default_ipv4.address) }} \ |
There was a problem hiding this comment.
fix typo in variable name: kube_api_address -> kube_api_host
| file: tasks/k8s/install-containerd.yml | ||
| apply: | ||
| tags: | ||
| - containerd-setup |
| - name: Temporary UFW configuration for Kubernetes installation | ||
| hosts: master | ||
| become: yes | ||
| tags: k8s | ||
| tasks: | ||
| - name: "Setup UFW with allow by default" | ||
| ufw: | ||
| state: enabled | ||
| default: allow | ||
| direction: incoming |
There was a problem hiding this comment.
Temporary disable firewall to allow safe provision
There was a problem hiding this comment.
Pull request overview
This PR hardens Kubernetes firewall exposure and simplifies configuration by deriving cluster node IPs from the control plane instead of requiring an operator-provided node CIDR.
Changes:
- Restricts Kubernetes API access via UFW rules built from discovered node InternalIPs + optional allowed CIDRs.
- Removes
KUBE_CLUSTER_NODE_CIDRplumbing and introducescluster_communication_portsfor node-to-node firewall rules. - Adds kubelet node IP configuration and updates kubeadm init to use
kube_api_host.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| infrastructure/server-setup/tasks/k8s/ufw.yml | Builds UFW allowlists from Kubernetes node IPs and applies per-port/per-node rules. |
| infrastructure/server-setup/tasks/k8s/init-master.yml | Updates kubeadm init arguments to use kube_api_host. |
| infrastructure/server-setup/tasks/k8s/configure-private-ip.yml | Adds kubelet --node-ip configuration and restarts kubelet. |
| infrastructure/server-setup/k8s.yml | Retags k8s install tasks and introduces a temporary UFW stage during provisioning. |
| infrastructure/server-setup/group_vars/all.yml | Replaces node CIDR variable with cluster_communication_ports list. |
| infrastructure/environments/templates/inventory/production.example.yml | Updates inventory documentation to reference kube_api_host. |
| infrastructure/environments/templates/inventory/inventory.template.yml | Updates inventory template documentation to reference kube_api_host. |
| infrastructure/environments/questions.ts | Removes KUBE_CLUSTER_NODE_CIDR question and changes default for API allowed CIDRs prompt. |
| .github/workflows/provision.yml | Removes KUBE_CLUSTER_NODE_CIDR propagation and adds kube_api_host to extra vars. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
makelicious
left a comment
There was a problem hiding this comment.
Looks good, thanks! Some suggestions to simplify the variable assignments. See if they are actionable.
| ufw: | ||
| rule: allow | ||
| port: "6443" | ||
| port: "22" |
There was a problem hiding this comment.
is there way to configure ssh port? I have seen variety in that, and I think previously there was
| ufw: | ||
| rule: allow | ||
| port: "22" | ||
| port: "6443" |
|
|
||
| - name: Build Kubernetes API allowed CIDRs | ||
| set_fact: | ||
| kubernetes_api_allowed_sources: >- |
There was a problem hiding this comment.
could we keep the ips / cidrs in the variable? We do explicit transformation from ip to cidr so it seems importnat.
There was a problem hiding this comment.
ah we combine these two
| - name: "Allow ssh connections" | ||
| - name: Build cluster node IP list | ||
| set_fact: | ||
| cluster_node_ips: >- |
There was a problem hiding this comment.
Would it make sense to have them as cidrs here already?
There was a problem hiding this comment.
This is particular IPs for cluster nodes (master and workers).
We are allowing node-to-node communication
| from_ip: "{{ kube_cluster_node_cidr }}" | ||
| port: "{{ item.0.port }}" | ||
| proto: "{{ item.0.proto }}" | ||
| from_ip: "{{ item.1 }}/32" |
There was a problem hiding this comment.
we map through cluster_node_ips again need to add /32. If it is needed it is okay. We could just define it once for clarity if it is possible.
There was a problem hiding this comment.
This rule is written specifically for cluster internal node-to-node communication.
Subnets are not part of this rule.
* Added integration environemnt * fix: Added mosip integration * fix: Repo link for helm chart * fix: Disable authentication * fix: Added MOSIP variables * Add verifiable credentials URLs to values.override.yaml * add Pyry to users * Update Verifiable Credentials URLs in values.override.yaml use proxy instead of the direct issuer - it adds issuerKey * chore: remove webhooks from infra * configure public env variables for MOSIP Collab (#2) * configure public env variables for MOSIP Collab * add mosip api secrets to deployment * move secrets / variables to appropriate places * bump version * disable not needed mocks * update esignet urls * Update Verifiable Credentials URLs in values.override.yaml use proxy instead of the direct issuer - it adds issuerKey --------- Co-authored-by: Vadym Mudryi <vadym.mudryi@gmail.com> * bump helm charts to v2.0.0 (#3) * fix: Add portal deployment * fix: Add portal deployment * fix: Add portal deployment * fix: name * add reference data related secrets and mappings * fix: Add performance testing CI/CD * fix: Release name * fix: Added environment * fix: Added environment * fix: Added (tmp) prod environment for perf-testing * fix: Remove file before downloading new version * fix: Added MOSIP integration * fix: Added environment file * fix: checkout before running * fix: hardcode output to elasticsearch * fix: smoke -> load * fix: Switch develop branch to v2.0.0 * fix: Bump version * chore: remove opencrvs/ocrvs-notification from image size comparison script * chore: remove opencrvs/ocrvs-metrics from image size comparison script * initial commit demo environment * fix: Add extra labels to runner * fix: Add self hosted runner additional labels variables * fix: Add depedendencies * fix: Add common demo reset-2fa * add demo env configuration files * fix: use custom path to store data * fix: use custom path to store data * demo staging * fix: Add staging opencrvs environment * fix: Added mosip integration * test * fix: Proper way to create tls secret * fix: Commit not-tested fix; * fix: Add implicit dependency pigz * fix: Added missed variables * fix: Add legacy migration after seeding users * fix: Run legacy migrations as part of cleanup * fix: Add opencrvs environment * fix: name * fix: Upgrade upload/download artifact actions * fix: Add CSP variable * fix: Disable mosip * fix: bypass scale up/down traefik * fix: Hardcoded path to MOSIP configuration file * fix: Added mosip integration * added missed building blocks * fix: Switch to mosip mock integration * fix: Move integration configuration options * fix: Hardcoded path to MOSIP configuration file * fix: Moved MOSIP to dedicated workflow * fix: Moved MOSIP to dedicated workflow * fix: job dependency * fix: job dependency * fix: Bump chart version * fix: Added backup env * fix: Enable restore on staging * fix: Dropped custom mosip variables * fix: add APM server from demo env * fix: MOSIP URLs * fix: Added tool for secret management * fix: Added description * fix: Disable reindex on depoy * disable notifications * feat: Add generic workflow to deploy any release * fix: Drop duplicate redirect * fix: remove duplicated keys * fix: Init beta release * fix: Move postgres to worker node * fix: Use platform.* * Update MOSIP_WEBSUB_CALLBACK_URL for integration * fix: Content security policy * fix: Content security policy * Update CONTENT_SECURITY_POLICY_WILDCARD for client env use wildcard due to needing minio.* for example. * force mosip-api image to v2.0.0-demo.1 * bump mosip-api image to 2.0.0-demo.2 * bump mosip-api image to 2.0.0-demo.3 * Update mosip_api image tag to 2.0.0-demo.4 * Update MOSIP API image tag to 2.0.0-demo.5 * fix: drop mistakenly added wildcard * fix: BGP rules for calico * fix: BGP rules for calico * fix: Use platform.* * fix: bypass scale up/down traefik * fix: Add CSP variable * fix: Proper way to create tls secret * fix: Add implicit dependency pigz * fix: Add legacy migration after seeding users * fix: Upgrade upload/download artifact actions * chore: add jamil as an ssh user on demo server * chore: add markus as an ssh user on demo server * feat: Init release * fix: Added changelog * fix: Token reference * fix: Token reference * chore: update version to 1.9.14 * docs: update changelog for 1.9.14 release candidate * fix: Skip configuration files override on environment:init (upgrade) flow (#306) * fix: BGP rules for calico * fix: Don't ask "Do you want to enable disk encryption?" for configured environment (#307) * fix: Adjust message to be shown only for disabled encryption * fix: Don't ask "Do you want to enable disk encryption?" for configured environment (#307) * fix: Adjust message to be shown only for disabled encryption * fix: Added changelog * TODO/FIXME * chore: Kubernetes upgrade from 1.34 to 1.35 (#310) * fix: Added changelog * chore: Kubernetes upgrade from 1.34 to 1.35 * fix: Duplicate key "when" * fix: Hardcoded cert-manager * fix: Upgrade calico according to official doc * fix: Adding missed "quotes" * fix: added Testing outbound HTTPS connectivity instead of ping * fix: Changelog * fix: Runner directory must belong to provision user (#313) * fix: Runner directory must belong to provision user (#313) * Add retry to curl command * Update mosip_api image tag to 2.0.0-demo.6 * Update MOSIP API image tag to 2.0.0-demo.7 * fix: Set string variable type * fix: Set string variable type * remove ocrvs-user-mgnt * chore: update version to 1.9.15 * docs: update changelog for 1.9.15 release candidate * fix: Add handler for not initialised git repository * fix: Enable Typha * fix: Remove mistakenly merged files (#319) * fix: Small amend to properly enable 2FA * fix: Git rid of call to countryconfig from elastalert service (#321) * Remove auth.* public subdomain — auth is now only reachable via gateway proxy (#320) * fix: Refactor Kubernetes & Runtime Questions (#323) * fix: Refactor Kubernetes & Runtime Questions * fix: Consistency in variable names * fix: Added network CIDRs validation * fix: Question clearification * fix: Variable name * fix: Revisited unattended upgrades policy (#324) * fix: Revisited unattended upgrades policy * Remove previously managed file * fix: Added placeholder value to avoid parsing issues * fix: Security adjustments (#326) * fix: Security adjustments * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix: Always set ansible_default_ipv4 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix: Amend question * drop man file * drop man file * fix: Version convention fix * fix: Init release script fix * chore: update version to 1.9.16 * docs: update changelog for 1.9.16 release candidate * fix: Release versioning (#337) * fix: Always restart self-hosted runner on k8s * fix: added Testing outbound HTTPS connectivity instead of ping (#338) * fix: Versions * agents.md / settings.json for infra * add copilot + codex config * link to docs * testing (#342) * feat: Backup migration script (#340) * fix: Add ACTIVATE_USERS to data_seed job (like for deploy)" * fix: Downgrade helm to 3.x version (#350) * feat: Added k9s to default userspace configuration * fix: Run differencial backup as non-root user after upgrade pgbackrest to v2.59.0 (#360) * fix: Make pgbackrest package version depending on distribution selection (#364) (#365) * fix: Retry connectivity check on transient failures in bootstrap script * fix: Retry runner archive download on transient failures * bump package version * ci: accept target_branch from core for older-channel releases - default master, so manual runs behave as before - skip the develop merge-back when targeting a release branch Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update version to 2.0.2 * docs: update changelog for 2.0.2 release candidate * fix: Added hold around pgbackrest to avoid upgrade/downgrade (#375) * ci: merge older-channel releases up instead of targeting master Mirrors opencrvs-core's init-release change, driven by the target_branch core passes down. When the target is a release branch rather than master this repo opens no release PR: it cuts merge-back/release-<version>-into-<target> from the target, merges the release branch into it, and PRs that into the target. Conflicts confined to the CHANGELOG and the chart/image version lines in the deploy workflows resolve to the target channel's copy and are noted in the PR body; anything else aborts. A target branch that does not exist yet is cut from master and given the same chart version bump and CHANGELOG section a normal init-release run would have written. Merge-back branch names now strip dots, matching the other repos. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: require the target release branch to exist Replaces the step that cut and initialised a missing target branch with a check that fails before anything is created. That release writes its own version bump and CHANGELOG, and a placeholder branch would make its own init-release run skip them. opencrvs-core resolves the target — now the next unreleased patch of the channel directly above the release's own — and passes it down; this repo only verifies it is there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: tidy up init-release and trim its comments Quality pass over the release workflow, no behaviour change: - one `Configure the git identity` step instead of the same two git config lines in two places - a `create_pr` helper instead of the "list, then create unless open" block written out twice - drop the `release_pr` output and the `target_slug` variable — both restate the target branch — and use bash parameter expansion for the dot-stripping instead of `echo | tr` subshells - resolve the expected merge conflicts with one `git checkout --ours` over the pathspec, staging just those files rather than `git add -A` - one `git fetch` for both refs the older-channel path needs Comments are about a quarter shorter: the header is a plain list of what the workflow does, and each step comment gives the reason rather than restating the code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: don't merge in CI when creating the merge-back branch Merging the release branch into the target during CI aborts the release init whenever the two channels disagree — core's 1.9.17 run died on modify/delete conflicts for packages 2.x dropped, before any PR, companion dispatch or draft release happened. The merge-back branch is now a copy of release/<version> whichever channel it targets, the same as the merge-back to develop has always worked: the branch exists so that the commits resolving conflicts with the target stay off the release branch, and whoever merges the PR does that resolution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: cut the merge-back from the target again, without blocking the release Restores the branch shape core's workflow was designed around: for an older channel the merge-back branch is cut from the target and the release branch merged into it, so the PR diff is only what that channel adds. What broke core's 1.9.17 run is fixed instead of avoided: a conflict outside the version/CHANGELOG class abandons the merge, skips the branch and its PR, and writes the job summary rather than failing the run; and modify/delete conflicts now resolve, keeping the target's deletions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ocrvs-13460): remove Sentry provisioning Core no longer reads SENTRY_DSN, so drop the Sentry DSN prompt from environment setup and the sentry.io reachability check from bootstrap. * chore: update version to 1.9.17 * docs: update changelog for 1.9.17 release candidate * docs(ocrvs-13460): note the Sentry removal * docs(ocrvs-13460): trim the changelog entry to the actionable part * ci: simplify init-release in step with core Same pass core just had, minus the parts that do not apply here — core still resolves the target branch and passes it in as target_branch. - VERSION and RELEASE_BRANCH as workflow-level env, so the version stops being interpolated into every command - the separate "check if release branch exists" step folded into the branch creation as an early exit - the draft release and its tag deletion are one step, since the second only undid what the first had just done - shorter names (BRANCH, TARGET), CHANGELOG.new + mv instead of the copy dance, and comments cut to why rather than what Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: Make provision user and group ids not rely on hardcoded values (#379) * fix: Make provision user and group ids not rely on hardcoded values * fix: Local provision scripts * chore: update version to 1.9.18 * docs: update changelog for 1.9.18 release candidate --------- Co-authored-by: Vadym Mudryi <vadym.mudryi@gmail.com> Co-authored-by: Pyry Rouvila <pyry.rouvila@gmail.com> Co-authored-by: jamil314 <jamil31415926@gmail.com> Co-authored-by: Pankaj Pant <44195004+pankaj-pant@users.noreply.github.com> Co-authored-by: Nil20 <alamnileem11@gmail.com> Co-authored-by: Md. Ashikul Alam <32668488+Nil20@users.noreply.github.com> Co-authored-by: Pyry Rouvila <pyry@opencrvs.org> Co-authored-by: Markus Laurila <makelicious@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: cibelius <cihan.m.bebek@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Oni <ocq1003@gmail.com> Co-authored-by: Riku Rouvila <riku.rouvila@gmail.com> Co-authored-by: tahmidrahman-dsi <tahmidrahman.dsi@gmail.com> Co-authored-by: Tahmid Rahman <42269993+tahmidrahman-dsi@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Description
Block etcd from access outside of the master node
etcd is listening on loopback and default ethernet*. Kubernetes API server talks to etcd over loopback. Second network interface is needed only for communication with other etcd's (cluster mode). OpenCRVS does support single master deployment only and there is no need to keep etcd ports open on default ethernet adapter.
( * ) - default ethernet adapter is always same as Kubernetes API host (KUBE_API_HOST)
Block Kubernetes API server access
By default access to Kubernetes API server is allowed from Kubernetes cluster nodes and approved admin/operator networks.
Drop KUBE_CLUSTER_NODE_CIDR variable
KUBE_CLUSTER_NODE_CIDR was introduced as part of PR #323
Propose of this variable was to allow operator define subnet for Kubernetes worker nodes.
Property is now taken from Kubernetes API (
kubectl get nodes) instead, seeGet Kubernetes nodes from control planeansible taskTesting
Infrastructure for testing:
Test scenarios:
Test were performed on following repo: https://github.com/adskyiproger/opencrvs-infrastructure/tree/fix-fw-rules
To reduce configuration time, snapshots were created:

Deployment on new servers
Configuration:
Provision workflow: ✅ https://github.com/adskyiproger/opencrvs-infrastructure/actions/runs/27610681084
Firewall state:

Nodes:
Since KUBE_API_HOST was not, Kubernetes use first available ethernet as advertise IP.

Add new node to the cluster: Update existing environment
Configuration:
Provision workflow: ✅ https://github.com/adskyiproger/opencrvs-infrastructure/actions/runs/27611734601/job/81637583208
Deploy cluster on private subnet
Configuration:
Provision workflow: ✅ https://github.com/adskyiproger/opencrvs-infrastructure/actions/runs/27618236712/job/81659661361
Environment init script was re-run and private KUBE_API_HOST was provided:

Private IP was identified properly:
KUBE_API_HOSTvalue for master nodeansible_hostfor workersFirewall status after provision:

Nodes:

Update cluster with approved networks on private subnet
Configuration:
Provision workflow: ✅ https://github.com/adskyiproger/opencrvs-infrastructure/actions/runs/27619020206/job/81662478578
Environment init was reconfigured:

Firewall:
NOTE: Additional run due to small bug which I fixes later: https://github.com/adskyiproger/opencrvs-infrastructure/actions/runs/27620013805/job/81666085518
Nodes:
