Skip to content

Credentials directory reader and AMBER_ overrides for custom sections (M-10a) - #1423

Draft
crimson-knight wants to merge 3 commits into
v2-devfrom
feature/v2-credentials-directory-reader
Draft

crimson-knight wants to merge 3 commits into
v2-devfrom
feature/v2-credentials-directory-reader

Conversation

@crimson-knight

Copy link
Copy Markdown
Member

What changed

Milestone M-10a, the Amber credentials reader (plan: followup-34-mica-beryl-amber-v2-plan.md, sections 6 and 7).

  • Amber::Configuration::CredentialsDirectory (src/amber/configuration/credentials_directory.cr) reads a systemd-style $CREDENTIALS_DIRECTORY, the layout LoadCredential= and LoadCredentialEncrypted= produce: one file per credential, with the file name as the key and the contents as the value. It takes the directory path as an argument and reads files only. It never reads ENV and never logs.
    • read returns every entry, read_credential / read_credential? read one entry, read_for(Type) reads only the files a section type declares, and read_into(Type) builds the typed YAML::Serializable section.
    • Each file is opened with O_NOFOLLOW | O_NONBLOCK | O_CLOEXEC, and its type and size are checked on the open descriptor. A symlink is followed only when its target is a file directly inside the directory.
    • Caps: 64 KiB per file and 1 MiB for one read, counting names and contents. Both can be changed and both are range-checked.
    • Each failure raises its own named Amber::Exceptions::CredentialsError subclass, and its message names the credential but never its value: CredentialsDirectoryNotFound, CredentialsDirectoryUnreadable, CredentialPathTraversal, CredentialNotFound, CredentialSymlinkOutsideDirectory, CredentialNotARegularFile, CredentialUnreadable, CredentialTooLarge, CredentialsDirectoryTooLarge, CredentialsDoNotFitSection, CredentialsRegisteredAfterSettingsLoad.
  • Amber::Configuration.register_credentials(key, Type, directory, maximum_file_size:, maximum_total_size:) registers the section. If Amber.settings has already loaded, it raises CredentialsRegisteredAfterSettingsLoad. The loader also raises a ConfigurationError when the environment YAML defines the same section.
  • The AMBER_{SECTION}_{KEY} override pass (env_override.cr) now reaches registered custom sections, credentials sections included:
    • Values follow the field type. String stays text, Bool and Float parse the same way as in the built-in pass, and other types are read as plain YAML scalars.
    • A nested section that is nil or absent is created when one of its variables is set.
    • The section is rebuilt from the YAML or credentials it was loaded from, which AppConfig#custom_config_sources records, so fields the type does not serialize keep their values.
    • The loader now loads custom sections before it runs the override pass.
  • SectionField (:nodoc:) lists, at compile time, the YAML fields of a section type and what kind of value each holds.
  • Docs: docs/guides/configuration.md covers registration, the require order for generated apps (require "./credentials" right after "amber" and before config/database.cr), the reader's rules and how the layers take precedence. The CHANGELOG has an Unreleased entry.
  • The branch also carries the ci: pin GitHub Actions to full commit SHAs commit from chore(v2): clear the 25 compiler warnings so the build is GREEN #1421, byte for byte. The local dependency-pin gate refuses to push a branch whose workflow names movable tags. Both SHAs were checked against their tags (checkout v4.4.0 11d5960a, install-crystal v1.9.4 19ef8675). The commit merges cleanly with chore(v2): clear the 25 compiler warnings so the build is GREEN #1421.

shard.yml is unchanged and no dependencies were added.

How it was verified

  • Full suite with crystal-alpha spec: 2446 examples, 0 failures, 0 errors, 0 pending
  • Library build with crystal-alpha build --no-codegen src/amber.cr: 0 errors, 0 warnings
  • crystal-alpha tool format --check src spec: clean
  • Ameba (pinned commit cdd58b3) on every changed file: 0 failures
  • Build color: YELLOW on this branch alone. Compiling the specs prints the 25 deprecation warnings in spec/amber/validations/params_spec.cr that are already on v2-dev. This diff adds no warnings. With chore(v2): clear the 25 compiler warnings so the build is GREEN #1421 merged in a throwaway worktree the result is GREEN: 2447 examples, 0 failures, 0 errors, 0 pending, 0 warnings.
  • Mutation checks, where each mutation makes the named spec fail:
    • removing O_NONBLOCK fails the FIFO spec
    • writing String overrides as plain scalars fails the empty-string override spec
    • not creating absent nested sections fails the nil nested-section spec
    • (from the build stage) disabling the symlink guard or the custom override pass fails the matching specs

Codex review (gpt-6-sol, xhigh) and resolutions

Severity Finding Resolution
BLOCKER Path race: the entry was checked by path and then reopened by path, so a swap to a symlink after the check could read a file outside the directory Fixed. The file is opened with O_NOFOLLOW and checked through the descriptor (fstat). A symlink is followed only to a file directly inside the directory, so no component below the directory can be swapped. An entry that becomes a link raises CredentialSymlinkOutsideDirectory through ELOOP.
MAJOR A regular file swapped for a FIFO before the open blocks startup Fixed. O_NONBLOCK on open plus a regular-file check on the descriptor. Spec: FIFO raises CredentialNotARegularFile without blocking.
MAJOR read_into read and checked undeclared files, so an unrelated large file failed registration Fixed. read_for reads only the type's declared keys. Spec: an undeclared oversized file and a subdirectory are ignored.
MAJOR No cap across files Fixed. Adds maximum_total_size (default 1 MiB, systemd's total) counting names and contents, enforced while listing and reading, with CredentialsDirectoryTooLarge.
MAJOR Rebuilding from to_yaml erased ignore_serialize fields Fixed. Sections are rebuilt from the loaded YAML node or the credential values (custom_config_sources), and to_yaml is used only when there is no source. Spec covers this through the loader.
MAJOR An override could not reach a nested section that is nil or absent Fixed. Field types are known at compile time (SectionField), and a missing nested mapping is created when one of its variables is set. Specs cover both the created case and the left-nil case.
MAJOR Plain-scalar coercion turned "" or null into nil for String fields Fixed. String fields get double-quoted scalars. Spec covered.
MAJOR Bool "1" and Float "1" behaved differently from the built-in pass Fixed. Both use the built-in parsing (parse_bool, to_f?), and a value that is not a number raises a ConfigurationError that names the variable. Spec covered.
MINOR realpath permission errors and entries removed after listing escaped the named errors Fixed. These now map to CredentialsDirectoryUnreadable, CredentialUnreadable and CredentialNotFound.
MINOR Int64::MAX cap overflowed maximum_file_size + 1 Fixed. Both caps must be between 0 and Int32::MAX, otherwise ArgumentError. Spec covered.
MINOR The register_credentials macro stringified a variable name Fixed. It is now a method that takes Symbol | String.

Full Codex output is kept in the AgentC report folder (followup-37-unblocked-work/codex-amber-credentials-reader.md).

crystal-conventions rubric

# MAJOR item Status
1 Config/secrets in shard.yml n/a: shard.yml unchanged
2 Config outside the library's config API Fixed: uses Amber's custom-section registry and typed YAML::Serializable sections. ENV is read only by the app initializer and the existing AMBER_* pass.
3 JSON perimeter n/a: no JSON
4 not_nil! / silencing .as(T) Fixed: no not_nil!. The one .as(YAML::Serializable) widens the builder's return type and is not a narrowing cast.
5 Toolchain / YELLOW reported as clean Fixed: crystal-alpha only. YELLOW is reported as YELLOW (pre-existing warnings, cleared by #1421).
6 Ruby-isms / leftover prints n/a
7 Log.setup, printing, deep ENV, shared mutable class vars Fixed: no logging, no printing, and ENV is read only in the documented override pass. Registry class variables are written once at boot while config files are required, and this is documented.
8 AED naming Fixed: list_of_ collections, question predicates (credential_section?, plain_file_name?), value_kind rather than kind
9 Bare rescue / no error base Fixed: specific rescues and the CredentialsError base
10 Specs for new behavior Fixed: every new path has specs, mutation-checked
11 Formatted Fixed
12 Lock / pins n/a: no dependencies added. The CI actions are pinned by SHA.
13 case … in on Grant unions n/a
14 Grant misuse n/a
15a/15b Replaced app files / jobs / migrations n/a

MINOR items: public methods have type restrictions and return types, and internal API is marked :nodoc:.

Follow-ups outside this PR

  • amber_cli's generated config/application.cr must require config/credentials.cr right after "amber" and before config/database.cr. That change belongs to the amber_cli repo.
  • Spec stand-in values carry an agentc-placeholder: marker that a local secret-scan gate requires. Maintainers can decide whether to keep it.

…ions

Milestone M-10a. Amber::Configuration::CredentialsDirectory reads a
systemd-style $CREDENTIALS_DIRECTORY (one file per credential, file name
as key, contents as value) and builds a typed YAML::Serializable section
from it. Amber::Configuration.register_credentials registers that section
and raises CredentialsRegisteredAfterSettingsLoad when Amber.settings is
already loaded, so a misordered require fails at boot.

The reader takes the directory as an argument and never reads ENV. It
rejects path traversal, symlinks that leave the directory, non-regular
files, unreadable files and files over a size cap (64 KiB by default),
each with a named CredentialsError subclass whose message names the
credential but never its value. Values are bound as double-quoted
scalars so contents such as "null" stay strings.

The AMBER_{SECTION}_{KEY} override pass now reaches registered custom
sections, including nested mappings and nil-valued fields, using the
YAML keys captured from the registered type. The loader reads custom
sections before the override pass and rejects a YAML section that
duplicates a credentials section.

Docs cover registration, the require order for generated apps, the
reader's rules and the precedence of the layers.
Address review findings on the credentials reader:

- Open each credential with O_NOFOLLOW, O_NONBLOCK and O_CLOEXEC and
  check its type and size on the open descriptor, so an entry swapped for
  a symlink or a FIFO after it was listed is rejected instead of read or
  waited on. Follow a symlink only when its target is a file directly
  inside the directory.
- Read only the files a section's type declares (read_for/read_into),
  add read_credential?, a 1 MiB total cap (CredentialsDirectoryTooLarge)
  and range checks on both caps.
- Map realpath and listing access errors to the named error classes.
- Replace the register_credentials macro with a method that takes a
  Symbol or String key and keyword caps.

Make the AMBER_ override pass for custom sections follow the field type:
String values stay text (an empty value stays empty), Bool and Float use
the built-in parsing, nested sections that are nil or absent are created
when one of their variables is set, and a section is rebuilt from the YAML
or credentials it was loaded from, so fields the type does not serialize
keep their values.
actions/checkout@v4 and oprypin/install-crystal@v1 were movable refs that their
owners can repoint. Each is now pinned to the commit its ref points to today
(checkout v4.4.0, install-crystal v1.9.4), so CI runs the same code as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant