feat: add support for ephemeral-encryption-keys by piyush-jena · Pull Request #999 · bottlerocket-os/bottlerocket-core-kit

piyush-jena · 2026-08-06T17:02:04Z

Description of changes:

Adds support for ephemeral encryption keys. The encryption keys are placed in tmpfs (/run) instead of the keystore and deleted after unlock.
As part of this feature, we encrypt the private partition with LUKS2 cipher. Because the encryption key is temporary, we format the partition every boot (with the same parameters) and re-encrypt every boot.
The datastore inside of private partition is further encrypted with fscrypt. This requires an additional fs feature to be enabled (encrypt) which we pass in the previous step (or using tune2fs in case of data partition).
Data Partition is formatted with the default parameters which is why we use systemd-makefs. We force the reformat by running wipefs first. This deletes all fs related signatures tricking systemd-makefs into reformatting and avoiding the additional overhead of writing 0s to the entire partition.
Additionally we use image features environment variables in various places to choose keystore path or trick early-boot-config into populating the datastore every boot (because we format the private partition every boot).

Testing done:

The tmpfs keystore has SELinux label private_t

bash-5.2# ls -lZ /run/rottweiler
total 0
bash-5.2# ls -lZ /run | grep rottweiler
drwx------.  2 root   root   system_u:object_r:private_t:s0     40 Aug 12 16:22 rottweiler

Verified the new keystore doesn't contain the keys.

[root@admin]# sheltie
bash-5.2# ls -lia /run/rottweiler/
total 0
918 drwx------.  2 root root  40 Aug 10 23:13 .
  1 drwxr-xr-x. 21 root root 520 Aug 10 23:13 ..

Verified the filesystems and the directories are encrypted.

bash-5.2# lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
NAME                     TYPE  FSTYPE         MOUNTPOINT
nvme1n1                  disk
`-nvme1n1p1              part  crypto_LUKS
  `-BOTTLEROCKET-DATA    crypt xfs            /local
nvme0n1                  disk
|-nvme0n1p1              part
|-nvme0n1p2              part  vfat
|-nvme0n1p3              part  vfat           /boot
|-nvme0n1p4              part  erofs
|-nvme0n1p5              part  DM_verity_hash
|-nvme0n1p6              part
|-nvme0n1p7              part  crypto_LUKS
| `-BOTTLEROCKET-PRIVATE crypt ext4           /var/lib/bottlerocket
`-nvme0n1p8              part
bash-5.2# rottweiler check directory /.bottlerocket/datastore encrypted
directory '/.bottlerocket/datastore' is encrypted

Reboot (with user-data) works - used bootstrap commands as user data to change settings.motd and in every boot its the updated value
user-data

[settings.bootstrap-commands.mybootstrap]
commands = [["apiclient", "set", "settings.motd=hello"]]
essential = true
mode = "always"

output:

bash-5.2# apiclient get settings.motd
{
  "settings": {
    "motd": "hello"
  }
}

A second consequence of reformatting data partition is that the journal for previous boot is lost which was also checked.

Moving the EBS volume to a different instance makes it unreadable
lockdown mode works

bash-5.2# apiclient get settings.motd
{
  "settings": {
    "motd": "hello"
  }
}
bash-5.2# apiclient lockdown
01:21:33 [INFO] Lockdown completed
bash-5.2# apiclient get settings.motd
Failed to get settings: Failed GET request to '/?prefix=settings.motd': Status 500 when GETing /?prefix=settings.motd: Data store error during get_prefix 'settings.motd' for Live: Data store integrity violation at /var/lib/bottlerocket/datastore/current/live: Live datastore missing

(A)I went through journal and it didn't find any errors during all this experiments
Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

ginglis13 · 2026-08-14T22:13:05Z

+Conflicts=umount.target
+After=selinux-policy-files.service
+Wants=selinux-policy-files.service
+Before=encrypt-datastore.service encrypt-localfs.service encrypt-privatefs.service


Suggested change

Before=encrypt-datastore.service encrypt-localfs.service encrypt-privatefs.service

Before=encrypt-datastore.service encrypt-local-fs.service encrypt-private-fs.service

Similar error in the [Install] section below

ginglis13 · 2026-08-14T22:14:49Z

+    // We do not write the marker file in case ephemeral encryption keys feature is enabled. This
+    // makes sure that post a reboot we repopulate the user-data in the datastore.
+    if is_ephemeral_encryption_keys_enabled {
+        fs::write(MARKER_FILE, "").unwrap_or_else(|e| {
+            warn!("Failed to create marker file {MARKER_FILE}, may unexpectedly run again: {e}")
+        });
+    }


The comment doesn't match the logic here -

We do not write the marker file in case ephemeral encryption keys feature is enabled.

if is_ephemeral_encryption_keys_enabled -> write the file

ginglis13 · 2026-08-14T22:16:43Z

This file doesn't appear to be used

vigh-m · 2026-08-15T04:33:53Z

⬆️ Updated following the comment and retested:

[root@admin]# sheltie
bash-5.2# ls -lZ /run/rottweiler
total 0
bash-5.2# ls -lZ /run | grep rottweiler
drwx------.  2 root   root   system_u:object_r:private_t:s0     40 Aug 15 04:17 rottweiler
bash-5.2# ls -lia /run/rottweiler/
total 0
1 drwx------.  2 root root  40 Aug 15 04:17 .
1 drwxr-xr-x. 21 root root 500 Aug 15 04:17 ..
bash-5.2# lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
NAME                     TYPE  FSTYPE         MOUNTPOINT
nvme1n1                  disk
`-nvme1n1p1              part  crypto_LUKS
  `-BOTTLEROCKET-DATA    crypt xfs            /local
nvme0n1                  disk
|-nvme0n1p1              part
|-nvme0n1p2              part  vfat
|-nvme0n1p3              part  vfat           /boot
|-nvme0n1p4              part  erofs
|-nvme0n1p5              part  DM_verity_hash
|-nvme0n1p6              part
|-nvme0n1p7              part  crypto_LUKS
| `-BOTTLEROCKET-PRIVATE crypt ext4           /var/lib/bottlerocket
`-nvme0n1p8              part
bash-5.2# apiclient lockdown
04:19:26 [INFO] Lockdown completed
bash-5.2# apiclient get settings.motd
Failed to get settings: Failed GET request to '/?prefix=settings.motd': Status 500 when GETing /?prefix=settings.motd: Data store error during get_prefix 'settings.motd' for Live: Data store integrity violation at /var/lib/bottlerocket/datastore/current/live: Live datastore missing
bash-5.2# systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

Signed-off-by: Piyush Jena <jepiyush@amazon.com>

Signed-off-by: Piyush Jena <jepiyush@amazon.com> Co-authored-by: Vighnesh Maheshwari <vighmah@amazon.com>

piyush-jena · 2026-08-17T20:10:36Z

Removed early-boot-config change in the force push because private partition is recreated. WIth the latest twoliter and kernel-kit (built from develop), fips works as well

bash-5.2# ls -lZ /run/rottweiler
total 0
bash-5.2# ls -lZ /run | grep rottweiler
drwx------.  2 root   root   system_u:object_r:private_t:s0     40 Aug 17 20:05 rottweiler
bash-5.2# lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT
NAME                     TYPE  FSTYPE         MOUNTPOINT
nvme0n1                  disk
|-nvme0n1p1              part
|-nvme0n1p2              part  vfat
|-nvme0n1p3              part  vfat           /boot
|-nvme0n1p4              part  erofs
|-nvme0n1p5              part  DM_verity_hash
|-nvme0n1p6              part
|-nvme0n1p7              part  crypto_LUKS
| `-BOTTLEROCKET-PRIVATE crypt ext4           /var/lib/bottlerocket
`-nvme0n1p8              part
nvme1n1                  disk
`-nvme1n1p1              part  crypto_LUKS
  `-BOTTLEROCKET-DATA    crypt xfs            /local
bash-5.2# rottweiler check directory /.bottlerocket/datastore encrypted
directory '/.bottlerocket/datastore' is encrypted
bash-5.2# apiclient get settings.motd
{
  "settings": {
    "motd": "hello"
  }
}
bash-5.2# apiclient report fips
Benchmark name:  FIPS Security Policy
Version:         v1.0.0
Reference:       https://csrc.nist.gov/
Benchmark level: 1
Start time:      2026-08-17T20:07:55.010321100Z

[PASS] 1.0       FIPS mode is enabled. (Automatic)
[PASS] 1.1       FIPS module is Amazon Linux 2023 Kernel Cryptographic API. (Automatic)
[PASS] 1.2       FIPS self-tests passed. (Automatic)

Passed:          3
Failed:          0
Skipped:         0
Total checks:    3

Compliance check result: PASS
bash-5.2# systemctl status encrypt-local-fs.service
● encrypt-local-fs.service - Encrypt Local Filesystem (/local)
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/encrypt-local-fs.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf, 00-fips-go.conf, 10-requires-tmp.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/encrypt-local-fs.service.d
             └─20-ephemeral.conf
     Active: active (exited) since Mon 2026-08-17 20:05:54 UTC; 2min 24s ago
 Invocation: b86902673df749078bd77b1e3bf01b75
   Main PID: 1650 (code=exited, status=0/SUCCESS)
   Mem peak: 2.5M
        CPU: 443ms

Aug 17 20:05:53 localhost systemd[1]: Starting Encrypt Local Filesystem (/local)...
Aug 17 20:05:54 localhost systemd[1]: Finished Encrypt Local Filesystem (/local).
bash-5.2# journalctl | head -5
Aug 17 20:05:51 localhost kernel: Linux version 6.18.39 (builder@buildkitsandbox) (x86_64-bottlerocket-linux-gnu-gcc (Buildroot 2025.05.1) 13.4.0, GNU ld (GNU Binutils) 2.44) #2 SMP PREEMPT_DYNAMIC Mon Aug 17 18:20:33 UTC 2026
Aug 17 20:05:51 localhost kernel: Command line: console=tty0 console=ttyS0,115200n8 net.ifnames=0 netdog.default-interface=eth0:dhcp4,dhcp6? quiet fips=1 module_blacklist=i8042 SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_BURST=25 module_blacklist=i8042 fips=1 init.systemd.unit=fipscheck.target root=/dev/dm-0 rootwait ro raid=noautodetect random.trust_cpu=on selinux=1 enforcing=1dm-mod.create="root,,,ro,0 661264 verity 1 PARTUUID=D52978BD-EF33-445C-9444-CFFD7748FC2D/PARTNROFF=1 PARTUUID=D52978BD-EF33-445C-9444-CFFD7748FC2D/PARTNROFF=2 4096 4096 82658 1 sha256 577af503ec391ccdcc1738294cbd37287392152a0b57e356ebfb73ee4cfd12fa 7c42a7fe3dcb190463029c6998e50628f0bda2c81f03dadf895daf6fe51c74fd 2 restart_on_corruption ignore_zero_blocks" -- systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
Aug 17 20:05:51 localhost kernel: KASLR enabled
Aug 17 20:05:51 localhost kernel: BIOS-provided physical RAM map:
Aug 17 20:05:51 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
bash-5.2# journalctl -u bootstrap-commands
Aug 17 20:05:56 ip-172-31-28-220.us-west-2.compute.internal systemd[1]: Starting Bootstrap Commands...
Aug 17 20:05:56 ip-172-31-28-220.us-west-2.compute.internal bootstrap-commands[2004]: 20:05:56 [INFO] Processing bootstrap command 'mybootstrap' ...
Aug 17 20:05:56 ip-172-31-28-220.us-west-2.compute.internal bootstrap-commands[2004]: 20:05:56 [INFO] Successfully ran bootstrap command 'mybootstrap'
Aug 17 20:05:56 ip-172-31-28-220.us-west-2.compute.internal systemd[1]: Finished Bootstrap Commands.

In the first boot we get the following journal logs. This shows that the bootstrap-commands run every time.

bash-5.2# journalctl | head -5
Aug 17 19:59:18 localhost kernel: Linux version 6.18.39 (builder@buildkitsandbox) (x86_64-bottlerocket-linux-gnu-gcc (Buildroot 2025.05.1) 13.4.0, GNU ld (GNU Binutils) 2.44) #2 SMP PREEMPT_DYNAMIC Mon Aug 17 18:20:33 UTC 2026
Aug 17 19:59:18 localhost kernel: Command line: console=tty0 console=ttyS0,115200n8 net.ifnames=0 netdog.default-interface=eth0:dhcp4,dhcp6? quiet fips=1 module_blacklist=i8042 SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_BURST=25 module_blacklist=i8042 fips=1 init.systemd.unit=fipscheck.target root=/dev/dm-0 rootwait ro raid=noautodetect random.trust_cpu=on selinux=1 enforcing=1dm-mod.create="root,,,ro,0 661264 verity 1 PARTUUID=D52978BD-EF33-445C-9444-CFFD7748FC2D/PARTNROFF=1 PARTUUID=D52978BD-EF33-445C-9444-CFFD7748FC2D/PARTNROFF=2 4096 4096 82658 1 sha256 577af503ec391ccdcc1738294cbd37287392152a0b57e356ebfb73ee4cfd12fa 7c42a7fe3dcb190463029c6998e50628f0bda2c81f03dadf895daf6fe51c74fd 2 restart_on_corruption ignore_zero_blocks" -- systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
Aug 17 19:59:18 localhost kernel: KASLR enabled
Aug 17 19:59:18 localhost kernel: BIOS-provided physical RAM map:
Aug 17 19:59:18 localhost kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable

piyush-jena · 2026-08-17T20:21:11Z

In aws-ecs-4 we see that once mode bootstrap-commands don't apply again

bash-5.2# journalctl -u bootstrap-commands
Aug 17 20:13:44 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: Starting Bootstrap Commands...
Aug 17 20:13:44 ip-172-31-42-194.us-west-2.compute.internal bootstrap-commands[1794]: 20:13:44 [INFO] Processing bootstrap command 'mybootstrap' ...
Aug 17 20:13:44 ip-172-31-42-194.us-west-2.compute.internal bootstrap-commands[1794]: 20:13:44 [INFO] Turning off bootstrap command 'mybootstrap'
Aug 17 20:13:44 ip-172-31-42-194.us-west-2.compute.internal bootstrap-commands[1794]: 20:13:44 [INFO] Successfully ran bootstrap command 'mybootstrap'
Aug 17 20:13:44 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: Finished Bootstrap Commands.
Aug 17 20:15:30 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: bootstrap-commands.service: Deactivated successfully.
Aug 17 20:15:30 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: Stopped Bootstrap Commands.
-- Boot 4b2fdec6fdd14cbc9d83708c8573e8ea --
Aug 17 20:15:38 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: Starting Bootstrap Commands...
Aug 17 20:15:38 ip-172-31-42-194.us-west-2.compute.internal bootstrap-commands[1695]: 20:15:38 [INFO] Bootstrap command mode for 'mybootstrap' is 'off'
Aug 17 20:15:38 ip-172-31-42-194.us-west-2.compute.internal systemd[1]: Finished Bootstrap Commands.

piyush-jena · 2026-08-17T22:06:35Z

Tried changing the settings in ecs-5-fips and post reboot I found that only settings through user-data persisted.
In case of ecs-4, they behaved as expected. Changes in settings persisted after reboots.

Signed-off-by: Piyush Jena <jepiyush@amazon.com> Co-authored-by: Vighnesh Maheshwari <vighmah@amazon.com>

piyush-jena · 2026-08-17T23:39:04Z

force push addresses the above comments.

jpculp reviewed Aug 6, 2026

View reviewed changes

Comment thread sources/rottweiler/src/key.rs

piyush-jena commented Aug 7, 2026

View reviewed changes

Comment thread packages/release/early-boot-config-ephemeral.conf Outdated

piyush-jena marked this pull request as draft August 7, 2026 21:13

piyush-jena force-pushed the ukis/core-kit branch 7 times, most recently from 5a99cea to 465d33c Compare August 12, 2026 16:36

jmt-lab requested changes Aug 12, 2026

View reviewed changes

piyush-jena force-pushed the ukis/core-kit branch 5 times, most recently from 99fd03a to 31bf6bb Compare August 14, 2026 07:43

piyush-jena mentioned this pull request Aug 14, 2026

Prepare to support UKIs #1007

Merged

arnaldo2792 marked this pull request as ready for review August 14, 2026 21:30

arnaldo2792 approved these changes Aug 14, 2026

View reviewed changes

ginglis13 reviewed Aug 14, 2026

View reviewed changes

arnaldo2792 self-requested a review August 14, 2026 22:48

jmt-lab approved these changes Aug 14, 2026

View reviewed changes

vigh-m force-pushed the ukis/core-kit branch from 31bf6bb to 3a8352c Compare August 15, 2026 04:32

piyush-jena and others added 2 commits August 17, 2026 17:38

bottlerocket-image-features: add ephemeral-encryption-keys

2e362d8

Signed-off-by: Piyush Jena <jepiyush@amazon.com>

rottweiler: add delete sub-command

4d8f640

Signed-off-by: Piyush Jena <jepiyush@amazon.com> Co-authored-by: Vighnesh Maheshwari <vighmah@amazon.com>

piyush-jena force-pushed the ukis/core-kit branch from 3a8352c to 700e8fa Compare August 17, 2026 18:31

arnaldo2792 reviewed Aug 17, 2026

View reviewed changes

release: add support for ephemeral-encryption-keys

0f24565

Signed-off-by: Piyush Jena <jepiyush@amazon.com> Co-authored-by: Vighnesh Maheshwari <vighmah@amazon.com>

piyush-jena force-pushed the ukis/core-kit branch from 700e8fa to 0f24565 Compare August 17, 2026 22:54

arnaldo2792 approved these changes Aug 18, 2026

View reviewed changes

arnaldo2792 merged commit aa4030a into bottlerocket-os:develop Aug 18, 2026
3 checks passed

	Before=encrypt-datastore.service encrypt-localfs.service encrypt-privatefs.service
	Before=encrypt-datastore.service encrypt-local-fs.service encrypt-private-fs.service

Conversation

piyush-jena commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ginglis13 Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

vigh-m Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

ginglis13 Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

ginglis13 Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

vigh-m commented Aug 15, 2026

Uh oh!

piyush-jena commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

piyush-jena commented Aug 17, 2026

Uh oh!

piyush-jena commented Aug 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

piyush-jena commented Aug 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

piyush-jena commented Aug 6, 2026 •

edited

Loading

piyush-jena commented Aug 17, 2026 •

edited

Loading