Skip to content

Security: bound request parameter parsing - #1416

Closed
crimson-knight wants to merge 10 commits into
amberframework:v2-devfrom
crimson-knight:fix/param-index-allocation
Closed

crimson-knight wants to merge 10 commits into
amberframework:v2-devfrom
crimson-knight:fix/param-index-allocation

Conversation

@crimson-knight

@crimson-knight crimson-knight commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Summary

Bounds how Amber parses request parameters. A request beyond a configured limit gets 400 Bad Request, and parsing never allocates in proportion to a number written in the request or to an unchecked body size.

  • Indexed keys (items[N]): the index uses a checked conversion and must be at most params.max_array_index (default 100) and at most params.max_array_index_gap (default 32) past the end of its array. Gap padding is capped per request by params.max_array_padding (default 1,000).
  • Other explicit defaults:
    • parameter count: 1,000
    • key nesting depth: 32, counted without splitting the key
    • key length: 1 KiB
    • field value length: 1 MiB
    • URL-encoded body: 8 MiB
    • multipart body: 128 MiB
    • multipart part headers: 16 KiB
    • one uploaded file: 64 MiB
    • JSON, XML and other document bodies read for schemas or params: 8 MiB
    • one WebSocket message, across its frames and fragments: 2 MiB
  • Streaming: FormScanner reads query strings and URL-encoded bodies byte by byte, with & and ; as separators. Raw key, value and body sizes are enforced as the bytes arrive; the query is read from the raw request target before URI parsing. Multipart bodies, part headers and files are counted while they stream, and the epilogue is drained within the body cap. Bounded reads go through a fixed scratch buffer and keep no more than their limit.
  • One budget per request: HTTP::Request#param_budget is shared by the query string, the router's form and multipart parses, and schema conversion. Cached router fields are charged once.
  • Bounded query and form readers: HTTP::Request#query_params checks the raw query before Crystal parses it, so every reader is bounded (schema contracts, WebSocket reconnection, the error page, application code). A rejected query raises on every access. HTTP::Request#form_params and #form_params? reuse Amber's bounded, cached form parse for any request method, so middleware and action reads are bounded and share one result with params.
  • Admission checks: every query, form and multipart pair is checked for key depth and bracket indices when it is admitted, so plain params readers see the same limits as schema contracts.
  • XML extraction: element#text keeps its base behavior (descendant text). Text and attribute values are measured in libxml's buffers before they are copied, checked against max_value_bytes and key limits, and charged to a total of four times max_document_body_bytes, XPath fields included. Documents are held to max_document_nodes (default 100,000): counted while they stream, and recovered malformed trees are counted before extraction.
  • Rejected bodies stay rejected: HTTP::Request#guard_body_read remembers a body rejection, so later params, form_params and schema reads raise it again instead of parsing the remaining bytes.
  • Media types: URL-encoded and multipart bodies are recognized by their media type, compared case-insensitively.
  • Configuration: limits live in a typed params: section (Amber::Configuration::ParamsConfig, AMBER_PARAMS_* overrides). Every limit has a range that keeps it clear of integer overflow.
  • Errors: Amber::Exceptions::BadRequest (status 400) covers malformed multipart framing and metadata; ParameterLimitExceeded is a BadRequest. Both are answered by the pipeline and the Error pipe. Schema contracts report them as parameter_limit_exceeded or invalid_request_body.
  • Uploads: each file is copied into a tempfile in one stable per-user 0700 directory (<tmpdir>/amber-uploads-<uid>, owner-checked, reused by every worker and restart). size is the bytes copied, not the client's declared size. During the action params.files[name].file is a readable handle, opened on first use. Amber closes it and deletes the tempfile when the request finishes; File#move_to(path) and File#retain! transfer ownership for later or asynchronous use, and cleanup skips an upload it no longer owns. A move across file systems copies to a file beside the destination and renames it into place, so a failed copy leaves both files intact. If another local user has claimed the shared upload directory, the process uses a new private directory with an unpredictable name. A duplicate part name deletes the replaced file at once, and a rejection deletes them all.
  • WebSocket gate: the pipeline detects upgrades the way the WebSocket handler does (case-insensitive Upgrade, Upgrade in Connection). A socket-only route no longer raises NilAssertionError: upgrades reach the handler, and a plain request gets 404.
  • WebSocket: socket routes read through Amber::WebSockets::BoundedProtocol, which counts each message across frames and fragments as they arrive and closes with 1009 when it passes the limit. The handshake query is checked before the 101 response. BinaryDecoder defaults max_frame_bytes: to the ingress limit and closes with 1009 when a smaller limit trips. It checks the size before and after Base64 decoding, and rejects length prefixes or field counts larger than the frame, plus any trailing bytes.
  • Nested indexed objects: items[0][name]=alice builds [{"name" => "alice"}] in JSONParser.

Configuration loading decisions

  • V2 files: the loader validates the effective params section after YAML and AMBER_PARAMS_* overrides, so an unusable limit fails at boot. It does not call the full AppConfig#validate!, because v2-dev's loader validates no section today. Turning that on would also start enforcing server, session, jobs, logging and mailer rules in existing apps, which is a separate change.
  • V1-format files: v2-dev still loads these as a supported compatibility path. Because these limits are a security boundary, the V1 path now carries a params: mapping and the same AMBER_PARAMS_* overrides, and validates them at load.

Compatibility

  • Forms with indices above 100, or sparse indices more than 32 past the current end of an array, now get a 400. Raise params.max_array_index / params.max_array_index_gap for those applications.
  • Uploads larger than 64 MiB, or multipart bodies larger than 128 MiB, get a 400. max_file_bytes accepts up to 1 GiB, because schema contracts hold uploads in memory.
  • Upload tempfiles now use a fixed amber-upload name with the client's plain extension. Amber deletes them when the request finishes; call move_to(path) or retain! during the action to keep one (for example before passing its path to a background job). File#file now opens the tempfile read-only.
  • A bracketed number above max_array_index is rejected for every params reader, not only schema contracts; forms that name records by id in brackets (users[1700][name]) need a higher max_array_index.
  • XML documents may hold at most params.max_document_nodes nodes (default 100,000).
  • FormData.parse, Multipart.parse and MultipartParser.parse_multipart_request take an optional ParamBudget (defaulting to the request's) where they previously took nothing or a ParamsConfig.
  • JSON and XML request bodies above 8 MiB get a 400; raise params.max_document_body_bytes for larger APIs.
  • WebSocket messages above 2 MiB close the connection with 1009; raise params.max_websocket_message_bytes if needed.

Tests

  • New specs cover each limit and each review finding. Every rejection asserts a 400 and a bound on bytes allocated (GC.stats.total_bytes). Request-path specs go through routing, the CSRF pipe and a schema contract, including ordinary query access and upload cleanup at the end of a request.
  • crystal-alpha spec: 2549 examples, 0 failures (base: 2402). Warnings are unchanged: the 25 existing Params#validation deprecation warnings in spec/amber/validations.
  • crystal-alpha tool format --check is clean.
  • Ameba (pinned cdd58b34) on the changed files: no new findings.
  • Codex review is not yet approved: the ninth pass still lists required changes, so this PR stays in draft.

Indexed parameter keys are converted with a checked conversion and must
fall within params.max_array_index and params.max_array_index_gap, with
gap padding capped per request by params.max_array_padding. Parameter
count, key nesting depth, key length, field value length, and URL-encoded
body size get explicit defaults. All limits live in the new typed
`params:` configuration section (AMBER_PARAMS_* overrides) and apply to
the router's form and multipart parsers and to the schema query, JSON
bracket, and multipart parsers.

A request beyond a limit raises Amber::Exceptions::ParameterLimitExceeded,
answered with 400 by the pipeline and the Error pipe, and reported as the
parameter_limit_exceeded code by schema contracts.

The WebSocket binary decoder now rejects length prefixes and field counts
larger than the message.
Addresses the review of the first parameter-limit commit:

- URL-encoded bodies and query strings are read pair by pair, so count,
  key, value and body-size limits reject before the body is held; both
  FormData.parse_part overloads and QueryParser.parse_query_string use
  the same checked reader, and router query access (Params#query,
  fetch_all) is checked once per request.
- Multipart bodies are capped by max_multipart_body_bytes and each file
  by max_file_bytes while streaming; the schema multipart paths read
  uploads through the same bounded policy, and one ParamBudget covers
  fields and files together.
- The WebSocket binary decoder caps the frame before Base64 decoding
  (max_frame_bytes, default 1 MiB) and rejects trailing bytes.
- JSONParser.parse_multipart counts and size-checks array elements.
- Gap and padding checks use subtraction, the count check compares
  before incrementing, and ParamsConfig#validate! enforces a range for
  every limit.
- The loader validates the effective params after YAML and AMBER_PARAMS_*
  overrides; the V1 settings path now carries `params:` and the same
  overrides with validation.
- A null-padded array slot is replaced with a container before
  descending, so items[0][name]=alice yields [{"name" => "alice"}].
Resolves the second review of the parameter-limit change:

- HTTP::Request#param_budget is shared by the query string, the router's
  form and multipart parses, and schema conversion; cached router fields
  are not charged twice. HTTP::Request#query_params checks the raw query
  before parsing, on every access until it succeeds, so schema, WebSocket
  and error-page readers are bounded too.
- URL-encoded reads never pass the remaining body allowance, and raw keys
  and values are size-checked before decoding. Key depth is counted
  without splitting, before any value is parsed.
- Multipart part headers are bounded by max_part_header_bytes while read
  (BoundedIO trims peeks to its allowance), the epilogue is drained within
  the body cap, and upload tempfiles are removed when a later part is
  rejected and when the request finishes. Tempfiles use a fixed name.
- max_file_bytes is limited to 1 GiB, which the schema's in-memory read
  can hold. The WebSocket binary decoder also checks the decoded frame size.
Resolves the third review:

- WebSocket routes read through Amber::WebSockets::BoundedProtocol, which
  caps each message across frames and fragments as they arrive
  (params.max_websocket_message_bytes, 2 MiB) and closes with 1009.
- Every multipart tempfile is tracked; a duplicate part name deletes the
  replaced tempfile at once, and a rejection deletes them all.
- Router::File copies through the handle File.tempfile created, closes it
  on success and failure, reports the bytes copied as `size`, and verifies
  the tempfile's identity before the schema reads it.
- FormScanner reads URL-encoded input byte by byte with & and ; as
  separators and enforces raw key, value and body sizes as bytes arrive.
- Malformed multipart framing or metadata raises
  Amber::Exceptions::BadRequest (400); ParameterLimitExceeded is now a
  BadRequest.
- The negative-index check applies only to bracketed segments.
- The query budget reads the raw request target before URI parsing, and
  method-override detection no longer copies the query.
Resolves the fourth review:

- ParamBudget#read_bounded reads through a fixed scratch buffer and keeps
  only permitted bytes, so an over-limit value never grows the result
  past its limit.
- JSON, XML and fallback document bodies (schema parser, JSONParser,
  XMLParser, router JSON params) are bounded by
  params.max_document_body_bytes (8 MiB).
- The WebSocket handler checks the handshake query before switching
  protocols.
- BinaryDecoder defaults to the ingress limit; FrameTooLargeError closes
  the socket with 1009.
- JSONParser array-form keys (`a[b][]`) get the same depth and index
  checks as nested keys.
- Upload tempfiles live in a private 0700 directory, and cleanup checks
  the tempfile's identity before deleting.
Fixes the fifth review's findings:

- HTTP::Request#form_params and #form_params? reuse Amber's bounded,
  cached form parse for every request method, so a middleware or action
  read of a URL-encoded body is bounded and shares one result.
- XML extraction reads each node's text once; every extracted key and
  value is checked (length, depth) and charged to a total of four times
  max_document_body_bytes, including XPath extraction.
- ParamBudget#admit! checks key shape and bracket indices, so query, form
  and multipart pairs are rejected before any params reader sees them.
- Uploads stay readable through the action: File#file opens the tempfile
  on first use, and Amber closes and deletes it after the response.
  File#move_to and File#retain! transfer ownership for later or
  asynchronous use; cleanup skips an upload it no longer owns.
- Context#websocket? matches the WebSocket handler (case-insensitive
  Upgrade, Upgrade in Connection). A socket-only route no longer raises
  NilAssertionError: upgrades reach the handler, plain requests get 404.
- Uploads use one stable per-user directory, <tmpdir>/amber-uploads-<uid>,
  mode 0700 and owner-checked, reused by every worker and restart.
Fixes the sixth review's in-scope findings:

- File#move_to across file systems copies to a new file beside the
  destination, renames it into place only after the copy succeeds, and
  deletes the tempfile only after the rename. A failed copy removes the
  partial file and leaves the destination and the tempfile intact.
- XML element#text again holds all descendant text, as in 76954f0.
  ExtractionAllowance measures each node's text once (with the whitespace
  strip would remove), checks max_value_bytes before building a value,
  charges every copy to the total cap, and copies each value once. XPath
  values get the same checks, and the element#text key is checked
  against the key limits.
- URL-encoded and multipart bodies are recognized by their media type,
  compared case-insensitively, in Params, method override detection and
  schema parsing.
Fixes the seventh review's in-scope findings:

- Params keeps a body rejection and raises it again on every later read,
  so a caller that catches it never sees a parse of the remaining bytes.
- XML values are written straight into a string of their stripped size,
  so each value is copied once and the charge matches the copy.
- New params.max_document_nodes (default 100,000): a parsed XML document
  is counted before extraction, which bounds extraction's caches.
- If another local user has claimed the shared upload directory, the
  process uses a new private directory with an unpredictable name.
- File.discard deletes a tempfile even when closing it fails, for both
  the upload copy and the cross-device staging file.
- A cross-device move that completed reports success even if removing
  the original tempfile fails; cleanup removes the leftover later.
… request

Fixes the eighth review's in-scope findings:

- A well-formed XML body is counted against max_document_nodes while it
  streams (XML::Reader), before libxml builds a tree; a recovered
  malformed document is counted by following libxml's links, with no
  per-node allocation, before extraction.
- ExtractionAllowance walks the tree by pointer and measures text and
  attribute values in libxml's buffers, decoding UTF-8 whitespace without
  allocating, so an over-limit value is rejected before it is copied.
- HTTP::Request#guard_body_read remembers a body rejection on the request;
  params, form_params, schema document reads (JSON, XML, fallback, CBOR,
  COSE) and schema multipart reads all raise it again.
- Upload cleanup closes and deletes every tempfile even when one handle
  fails to close or one file fails to delete, logging the failure.
@crimson-knight

Copy link
Copy Markdown
Member Author

Superseded by #1419, which carries these commits rebased onto #1418 (the Crystal 1.22 spec fix) plus the later review fixes. Closing this draft in favor of #1419.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant