[ykhrustalev]

Overview

A request with deeply nested JSON crashes llama-server. nlohmann parses it fine. The stack overflows later, in llama.cpp code that walks the parsed JSON with recursion:

request	where the stack overflows (gdb, Linux, master 620d222)
tool parameters schema, response_format schema	common_chat_schema_builder::build_node / build_array
tool-call arguments	jinja value_to_string_repr / value_array_t::as_string
/v1/systemone state	jinja value_to_json_internal (the template's tojson); deeper, decision_d1omni_escape

This PR checks the nesting depth before that code runs and answers 400 past 128 levels:

• common_chat_templates_apply: tools, tool-call arguments, chat_template_kwargs and json_schema, after they are parsed. This covers /v1/chat/completions, /v1/responses, /v1/messages and /apply-template.
• /completion and /v1/completions: json_schema (server-schema.cpp).
• /v1/systemone: the whole body (parse_questions).
• gemma4: a tool result parsed from text stays text if it is too deep.

common/json.cpp is not changed. The check is common_chat_check_json_depth in chat.cpp; it stops one level past the limit, so its own recursion is at most 129 calls deep.

Additional information

Proof 1, reproduction on master (crashes at d=1000 on macOS; use d=30000 on Linux):

llama-server -hf LiquidAI/LFM2.5-VL-450M-GGUF -hff LFM2.5-VL-450M-F16.gguf --port 8080 &
python3 -c "d=1000; s='{\"type\": \"array\", \"items\": '*d+'{\"type\": \"string\"}'+'}'*d; print('{\"messages\": [{\"role\": \"user\", \"content\": \"hi\"}], \"max_tokens\": 1, \"tools\": [{\"type\": \"function\", \"function\": {\"name\": \"f\", \"description\": \"d\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": '+s+'}}}}]}')" > deep.json
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary @deep.json

Proof 2, the same kinds of request, master and this PR (HTTP code per nesting depth, macOS M3 Ultra):

master sys:     100=200 120=200 130=200 700=500 1000=500 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
master tool:    100=200 120=200 130=200 700=DIED 1000=DIED 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
master rf:      100=200 120=200 130=200 700=DIED 1000=DIED 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
master strargs: 100=200 120=200 130=200 700=200 1000=200 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
this PR, all:   100=200 120=200 130=400 700=400 1000=400 2000=400 10000=400 30000=400 1000000=400

sys = /v1/systemone state (d1-omni), tool = tool parameters, rf = response_format, strargs = tool-call arguments as a string (as an object they give the same codes). master's 500 for sys at 700 and 1000 is the batch size error, not a crash. On Linux master dies from 10,000 (schemas) and 30,000 levels (state, arguments); this PR answers 400 from 130. At 100 and 120 levels the answers are the same as on master.

Proof 3, tests. test-chat gets test_chat_json_depth (the limit, and 1,000 levels in tool parameters, tool-call arguments, chat_template_kwargs and json_schema); test-server-component gets decision_depth. With master's code (without the three direct calls of the new function) both fail: test-chat because the 1,000-level inputs are not rejected, decision_depth with another error (too many options (2), this model supports at most 0). With this PR both pass on macOS and Linux.

• Cost (M3 Max): a 0.33 MB chat request 0.02 ms (parse 1.4 ms); 10 MB 0.54 ms (parse 35 ms); an array of 1M integers 3.8 ms (parse 41 ms).
• 128 levels: the deepest JSON in the repo's tests is 8 levels.

Not fixed here:

• json-schema-to-grammar recursion that does not come from nesting: long $ref chains, many optional properties, one long property name in an object with additionalProperties.
• macOS: some templates recurse in jinja macros once per level and overflow the 512 KB HTTP thread stack below 128 levels: IFM-K2-Horizon, GigaChat3, GigaChat3.1, Apertus-8B, llm-jp-4.1 and gpt-oss on tool parameters; gemma-4 on tool-call arguments, and on tool results with the interleaved gemma-4 template. In a Release build a limit of 64 would cover them, in a Debug build it would not.
• A request that ends with an assistant message (prefill) is not checked: its text goes to the PEG output parser, which recurses once per level (Eval bug: peg-parser: nested JSON input overflows the parser stack during chat parsing #29728). With response_format, 1,000 levels in the prefill still crash llama-server on macOS with this PR.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, distributed testing and implementation under a human supervision

[ngxson]

it should already be fixed upstream, we need to bump version: nlohmann/json#5389

[ykhrustalev]

@ngxson please re-open if needed

[ykhrustalev]

@ngxson #30253 does not fix this. Master f2cc728 includes #30253 and still crashes at the same depths as before it.

Lowest tested depth that crashes the server on master f2cc728 (tested 100, 120, 130, 700, 1,000, 2,000, 10,000, 30,000, 1,000,000):

request	macOS (M3 Ultra)	Linux x86
/v1/chat/completions, tool parameters schema	700	10,000
/v1/chat/completions, response_format schema	700	10,000
/v1/chat/completions, tool-call arguments	2,000	30,000
/v1/systemone, state (d1-omni)	2,000	30,000

The recursion that overflows the stack is in llama.cpp code. #30253 covers nlohmann's copy, dump() and compare.

gdb, Linux, master f2cc728, tool schema nested 10,000 levels (19,523 frames):

Thread 7 "llama-server" received signal SIGSEGV, Segmentation fault.
#0  _int_malloc
#1  __GI___libc_malloc
#2  operator new(unsigned long)
#3  nlohmann::basic_json<...>::basic_json(...)
#4  common_chat_schema_builder::build_node(common_json const&, std::string const&)
#5  common_chat_schema_builder::build_array(common_json const&, std::string const&)
#6  common_chat_schema_builder::build_node(common_json const&, std::string const&)
#7  common_chat_schema_builder::build_array(common_json const&, std::string const&)
...
#19518 httplib::Server::process_and_close_socket(int)
#19519 httplib::ThreadPool::worker(bool)
#19522 clone3

/v1/systemone state nested 30,000 levels (13,616 frames):

Thread 6 "llama-server" received signal SIGSEGV, Segmentation fault.
#0  nlohmann::basic_json<...>::basic_json(...)
#1  common_json::array()
#2  decision_d1omni_escape(common_json const&)
#3  decision_d1omni_escape(common_json const&)
#4  decision_d1omni_escape(common_json const&)
...
#13611 httplib::Server::process_and_close_socket(int)
#13612 httplib::ThreadPool::worker(bool)
#13615 clone3

Reproduction on master (d=1000 on macOS, d=10000 on Linux):

llama-server -hf LiquidAI/LFM2.5-VL-450M-GGUF:F16 --jinja --port 8080 &
python3 -c "d=1000; s='{\"type\": \"array\", \"items\": '*d+'{\"type\": \"string\"}'+'}'*d; print('{\"messages\": [{\"role\": \"user\", \"content\": \"hi\"}], \"max_tokens\": 1, \"tools\": [{\"type\": \"function\", \"function\": {\"name\": \"f\", \"description\": \"d\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": '+s+'}}}}]}')" > deep.json
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary @deep.json

With this branch rebased on f2cc728, the same requests get HTTP 400 from 130 levels (500 for tool-call arguments sent as a string) and the server stays up, on macOS and Linux.

GitHub does not let me reopen this PR: the open pull request limit is reached.

[ngxson]

if you think this is a valid bug in json parsing, better to just open a PR on upstream https://github.com/nlohmann/json and we will apply it as a patch if it gets merged

[ykhrustalev]

@ngxson I checked this.

The crash is in llama.cpp, not in nlohmann. nlohmann reads the deep JSON fine. The server dies later, in llama.cpp functions that call themselves once per level: common_chat_schema_builder::build_node and decision_d1omni_escape (gdb output above). A change in nlohmann/json cannot fix that, so llama.cpp still has to reject very deep JSON.

The nlohmann bug is already fixed upstream. nlohmann can report the depth while it parses (the parser callback), so llama.cpp does not need its own bracket counting. In 3.12.0 that callback is very slow on big inputs, which is why this PR counted brackets itself. nlohmann fixed it on develop in nlohmann/json#5457 (on top of nlohmann/json#5342). Both are merged, not released.

Parse time with the depth callback, Linux x86, -O3:

input	no callback	callback, 3.12.0 (master)	callback, with #5342 + #5457
200,000 objects, 10.6 MB	151 ms	19,044 ms	196 ms
2,000 chat messages, 464 KB	3.2 ms	5.4 ms	3.6 ms
1M integers, 6.9 MB	102 ms	115 ms	112 ms

New version of this PR, 2 commits, no own bracket counting:

1. vendor/nlohmann/json-callback-parser.patch: is an optimized communication language better for training and inference? #5342 and llava: remove prog parameter from ArgumentParser #5457 applied to 3.12.0, the same way as json-deep-nesting.patch, with a TODO to remove it after the next nlohmann release.
2. common_json::parse uses the callback to stop at level 129:

static bool json_check_depth(int depth, ordered_json::parse_event_t event, ordered_json & /*parsed*/) {
    const bool opens = event == ordered_json::parse_event_t::object_start ||
                       event == ordered_json::parse_event_t::array_start;
    if (opens && depth >= COMMON_JSON_MAX_DEPTH) {
        throw common_json_error("JSON is nested more than " + std::to_string(COMMON_JSON_MAX_DEPTH) + " levels deep");
    }
    return true;
}

common_json common_json::parse(const std::string & text) {
    ...
        as_json(&out) = ordered_json::parse(text, ordered_json::parser_callback_t(json_check_depth));
    ...
}

Evidence

• The patched callback parser is the same as nlohmann's at 1dc1d09fc, except two comment wordings and the JSON_DIAGNOSTIC_POSITIONS code of Slight quantization improvement for Q4_K and Q5_K #5361, which llama.cpp does not use. nlohmann's own test file from that commit, tests/src/unit-class_parser.cpp (one helper that uses a newer parse option is stubbed out):

  patched header:    [doctest] assertions: 9316 | 9316 passed | 0 failed |
  3.12.0 (master):   [doctest] assertions: 9316 | 9307 passed | 9 failed |
  

  The 9 failures on master are the cases is an optimized communication language better for training and inference? #5342 and llava: remove prog parameter from ArgumentParser #5457 added.

• Same results: 528,078 generated JSON texts up to 128 levels (duplicate and empty keys, escapes, big arrays of objects, invalid texts) give the same value or the same error with and without the callback, MISMATCHES 0 in all 14 runs.

• The requests from the table above, macOS, master and this version (HTTP code per nesting depth):

  master sys: 100=200 120=200 130=200 700=500 1000=500 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
  master tool: 100=200 120=200 130=200 700=DIED 1000=DIED 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
  master rf: 100=200 120=200 130=200 700=DIED 1000=DIED 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
  master strargs: 100=200 120=200 130=200 700=200 1000=200 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
  master objargs: 100=200 120=200 130=200 700=200 1000=200 2000=DIED 10000=DIED 30000=DIED 1000000=DIED
  fix sys: 100=200 120=200 130=400 700=400 1000=400 2000=400 10000=400 30000=400 1000000=400
  fix tool: 100=200 120=200 130=400 700=400 1000=400 2000=400 10000=400 30000=400 1000000=400
  fix rf: 100=200 120=200 130=400 700=400 1000=400 2000=400 10000=400 30000=400 1000000=400
  fix strargs: 100=200 120=200 130=500 700=500 1000=500 2000=500 10000=500 30000=500 1000000=500
  fix objargs: 100=200 120=200 130=400 700=400 1000=400 2000=400 10000=400 30000=400 1000000=400
  

  At 100 and 120 levels the answers are the same as on master. Linux gives the same "fix" lines.

I will update this PR with this version once I can reopen it (I am at the open PR limit).

[ngxson]

The crash is in llama.cpp, not in nlohmann. nlohmann reads the deep JSON fine. The server dies later, in llama.cpp functions that call themselves once per level: common_chat_schema_builder::build_node

not sure but sounds like the fix should land in the chat.cpp system, not json.cpp

common/json.cpp is just a thin wrapper around nlohmann, patches must never be added inside it

[ykhrustalev]

@ngxson Updated: common/json.cpp is not changed any more. The check is in the chat code, common_chat_check_json_depth in chat.cpp, called in common_chat_templates_apply after the tools, tool-call arguments, chat_template_kwargs and json_schema are parsed. The server calls it for the json_schema of /completion and for the /v1/systemone body. Past 128 levels the server answers 400; the description has the numbers.

Two points for you:

• 128 still lets some templates crash on macOS, because they recurse in jinja macros once per level (IFM-K2-Horizon, GigaChat3, Apertus, llm-jp and gpt-oss on tool parameters, gemma-4 on tool-call arguments). In a Release build 64 would cover them. Which limit do you want?
• common_chat_check_json_depth is public in chat.h because the server uses it too. Fine there?

[ServeurpersoCom]

LGTM, the check now lives in chat.cpp as requested. Confirmed on Linux that every path that crashed on master now returns a 400, and that #31032 and #31099 still segfault on a nested response_format schema. 128 sounds right to me, the macOS cases come from the HTTP thread stack size and can be a follow-up.

[ngxson]

I was not convinced with either fixes you proposed, so I pushed 7964e8d

It's a stronger fix because the root cause is that we don't have any bound for recursive functions.

Fix by adding a call-side validation works, but cost (1) walking the json tree twice and (2) won't establish a strong pattern for future code to follow

The code base may have many other places where recursive functions having no bound, will be just a simple prompt away from fixing them.

[aldehir]

I agree, doing a second parsing pass feels wrong and it won't protect against $ref loops in JSON schemas, so there would need to be a second guard in place when a recursion limit would address both.

[ServeurpersoCom]

Indeed, even better: bounding each recursive function fixes the root cause, avoids walking the JSON twice and also catches $ref loops. Retested on Linux: the same paths return a 400, 120 levels still work, and a $ref loop no longer crashes. The Windows failure is test-thread-safety, which fails the same way on master.