server: Add support for binding to multiple addresses by erusev · Pull Request #28690 · ggml-org/llama.cpp

erusev · 2026-09-10T08:46:42Z

Binding llama-server to a VPN address currently makes it unavailable through localhost, breaking clients that default to localhost. Supporting multiple addresses lets one server remain accessible through both.

--host now accepts a comma-separated list of IP addresses and Unix socket paths. The server creates one listener per address, all serving the same API and model through a shared HTTP worker pool.

TCP listeners share the configured port. With --port 0, the first TCP listener selects an available port for the others. Every address must bind successfully; otherwise, startup fails and closes the listeners already bound. Each listening URL is logged separately.

pwilkin · 2026-09-10T10:43:23Z

/bot review

ggml-gh-bot · 2026-09-10T10:43:26Z

Automated code review

Review of PR 28690: multiple `--host` binding

I reviewed the arg-parser changes, the server_http_context rework, and the tests. The overall approach is sound: per-address httplib::Server instances, one accept thread each, a shared ThreadPool drained in join(). I verified the httplib internals this relies on (bind_to_port binds and listens, Server::stop() is concurrency-safe via svr_sock_.exchange, the per-listen TaskQueue is owned and destroyed by listen_internal() so the no-op server_http_task_queue::shutdown() is correct, and route registration happens after init() so every server gets every route). No correctness bug found in the C++ threading/shutdown logic itself. Findings below.

Blocking

(point 1) tools/server/tests/unit/test_basic.py, test_server_multiple_addresses_bind_failure: this test will fail on the Windows CI runner. llama-server sets SO_REUSEADDR unconditionally on its listen sockets (server-http.cpp, set_socket_options), and on Windows SO_REUSEADDR permits binding an address:port that another socket is already actively bound to (unlike Linux, where it only allows rebinding from TIME_WAIT). So on windows-2025 the second listener will bind ::1:<port> successfully, the server will start, and pytest.raises(RuntimeError, match="Server process died") will fail. Since tests.sh runs pytest with -x, this aborts the entire Windows job. Add a skip for sys.platform == "win32" (the success and mixed-address tests are platform-independent and fine).

Will slow the review

(point 2) No linked issue. This is a user-facing server feature, and CONTRIBUTING expects features to start with an issue/discussion before a PR. Please link the relevant issue or open one so maintainers can gauge interest before spending review time here.

(point 3) tools/server/server.cpp:113 adds common_log_flush(common_log_main()) with no explanation, and it looks unrelated to multi-address binding. I assume it exists so the new failure-path SRV_ERR lines reach the --log-file before exit (the log singleton is deliberately leaked, common/log.cpp), which the new tests assert on. If that's the reason, please say so in the PR description; if not, drop it.

(point 4) common_params_to_map (common/arg.cpp:1226) collapses repeated --host to the last value in the preset map, while the real parser keeps the first and appends the rest. I traced the consumers: router children always override LLAMA_ARG_HOST with CHILD_ADDR (server-models.cpp), so there's no functional break today, but the semantic divergence is worth a line in the README's preset section. Relatedly, duplicate addresses (--host 127.0.0.1 --host 127.0.0.1) fail startup with only an opaque "couldn't bind" error; consider deduplicating or rejecting duplicates explicitly.

Nits

(point 5) tools/server/server-http.cpp:119: the pimpl->hosts.size() > 1 condition is loop-invariant, and validation of a later host runs after earlier listeners were already constructed. Hoist a single validation pass over pimpl->hosts above the server-creation loop.

(point 6) tools/server/server-http.cpp:511: the listener-start failure path doesn't clear listening_address, unlike the bind-failure path at line 483. Harmless today since the address is only logged after a successful start, but inconsistent.

(point 7) test_server_mixed_addresses hardcodes /tmp/llama.sock. It is never actually bound (validation fails first), but using tmp_path would avoid implying a real path is created. Also minor: with LLAMA_ARG_HOST set, each additional --host occurrence prints the "env variable will be overwritten" warning, which is only true for the first occurrence.

Everything else looks good: all params.hostname consumers were migrated, the --spec-type-style repeatable-arg precedent is followed consistently in both parse paths, router-mode/preset/env interactions behave as the README documents, and the test file's autouse fixture prevents the new server attributes from leaking into other tests.

This review was generated automatically by pi coding agent using zai-org/GLM-5.3. It may contain mistakes. Maintainers make the final call.

ngxson · 2026-09-10T14:02:54Z

+
+Each `--host` value is literal; commas are not separators. A single path ending in `.sock` keeps its existing Unix socket behavior, including commas in the path. Mixing TCP addresses and Unix socket paths is rejected in either order.
+
+`LLAMA_ARG_HOST` and the preset `host` setting each accept one literal address or path. The first CLI `--host` overrides that value; additional occurrences add addresses.


this behavior is inconsistent

ngxson

instead of repeating the --host arg, it's cleaner to allow CSV style input, we already had many other args doing this

example: --host 10.1.2.123,79.52.1.22,::1

and then simply throw error if we are mixing UNIX socket and IP, we should only support one or the other

erusev · 2026-09-10T14:54:30Z

Amended. --host now accepts comma-separated addresses.

ngxson · 2026-09-10T17:29:25Z

+        string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", string_join(params.hostnames, ",").c_str()),
        [](common_params & params, const std::string & value) {
-            params.hostname = value;
+            params.hostnames = string_split(value, ",");


use parse_csv_row

ngxson · 2026-09-10T17:29:36Z

    add_opt(common_arg(
        {"--host"}, "HOST",
-        string_format("ip address to listen, or bind to an UNIX socket if the address ends with .sock (default: %s)", params.hostname.c_str()),
+        string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", string_join(params.hostnames, ",").c_str()),


Suggested change

string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", string_join(params.hostnames, ",").c_str()),

string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", params.hostnames[0].c_str()),

ngxson · 2026-09-10T17:30:57Z

some test cases seem redundant

just one test case is generally enough for a simple feature like this

ngxson · 2026-09-10T17:33:58Z

+## Listening on multiple addresses
+
+Use `--host 127.0.0.1,::1` to listen on several addresses. Each address serves the same routes, including streaming responses. All addresses must bind successfully; otherwise startup fails and closes the sockets already bound.
+
+All addresses use `--port`. With `--port 0`, the first address selects an available port and the remaining addresses must bind that same port. The startup log lists every listening URL. HTTP workers share the existing `--threads-http` budget, with one accept thread per address.
+
+Unix socket paths ending in `.sock` keep their existing behavior. TCP addresses and Unix socket paths cannot be mixed.
+
+`LLAMA_ARG_HOST` and the preset `host` setting accept the same comma-separated format.


this whole section seems redundant, better to remove this

ngxson · 2026-09-10T17:36:20Z

+        if (!listening_address.empty()) {
+            listening_address += ", ";
+        }
+        listening_address += is_sock ? string_format("unix://%s", host.c_str())
+                                     : string_format("%s://%s:%d", is_ssl ? "https" : "http", common_http_format_host(host).c_str(), port);


looks quite overkill, let's just do it simple:

for (addr : addresses) { LOG_INF("Listening on: %s", addr.c_str()) }

ngxson · 2026-09-10T17:37:27Z

+    const bool use_unix_sockets = string_ends_with(pimpl->hosts.front(), ".sock");
+    for (const auto & host : pimpl->hosts) {
+        if (host.empty() || string_ends_with(host, ".sock") != use_unix_sockets) {
+            SRV_ERR("%s", "--host requires non-empty TCP addresses or Unix socket paths; TCP and Unix socket addresses cannot be mixed\n");


since we have different thread pool for each addr, maybe mixing TCP + UNIX is trivial after all?

mixed TCP/Unix sockets needed no new infrastructure, and testing passed in both address orders

Assisted-by: Codex

erusev · 2026-09-11T07:59:10Z

Amended.

…8690) Squash of ggml-org#28690 at f99f961. --host takes a comma-separated list of addresses / unix socket paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ngxson · 2026-09-21T11:05:53Z

sorry for the delay, there are still some minor issues that need to be addressed before merging, I will push them today or tmr

* Add support for binding llama-server to multiple addresses Assisted-by: Codex * remove redundant thread handler * make it clear about overlapping addr * reject --port 0 with multiple tcp addr * improve arg handler * nits * fix test * nits 2 * nits * nits 2 --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

erusev requested review from a team as code owners September 10, 2026 08:46

github-actions Bot added documentation Improvements or additions to documentation server labels Sep 10, 2026

erusev force-pushed the server-multiple-hosts branch 2 times, most recently from b8a4e1e to 0afc4ba Compare September 10, 2026 10:39

ngxson reviewed Sep 10, 2026

View reviewed changes

erusev force-pushed the server-multiple-hosts branch from 0afc4ba to f45e12c Compare September 10, 2026 14:51

ngxson reviewed Sep 10, 2026

View reviewed changes

Add support for binding llama-server to multiple addresses

5a1ffdb

Assisted-by: Codex

erusev force-pushed the server-multiple-hosts branch from f45e12c to 5a1ffdb Compare September 11, 2026 07:51

ngxson added 7 commits September 12, 2026 18:22

remove redundant thread handler

6f13048

make it clear about overlapping addr

8a710b6

reject --port 0 with multiple tcp addr

a0de4a8

improve arg handler

cb9901f

nits

90a0aea

fix test

0ba1e5e

nits 2

f99f961

ngxson changed the title ~~Add support for binding llama-server to multiple addresses~~ server: Add support for binding to multiple addresses Sep 13, 2026

Merge branch 'master' into server-multiple-hosts

860d4da

ngxson added 3 commits September 21, 2026 18:10

Merge branch 'master' into server-multiple-hosts

91276ed

nits

9b9be77

nits 2

9564b73

ngxson merged commit 217f81c into ggml-org:master Sep 22, 2026
17 checks passed

ggerganov added the highlight Changes that will be highlighted in the next release notes label Sep 23, 2026

This was referenced Sep 23, 2026

llama.cpp : bump version to 0.5.0 #29303

Closed

llama.cpp : bump version to 0.5.0 #29333

Merged

bernardladenthin mentioned this pull request Sep 27, 2026

llama.cpp b11080 → b11211, CUDA 13.4, ROCm 10 (TheRock), sccache on every Windows Ninja job bernardladenthin/java-llama.cpp#456

Merged

6 of 8 tasks


		Each `--host` value is literal; commas are not separators. A single path ending in `.sock` keeps its existing Unix socket behavior, including commas in the path. Mixing TCP addresses and Unix socket paths is rejected in either order.

		`LLAMA_ARG_HOST` and the preset `host` setting each accept one literal address or path. The first CLI `--host` overrides that value; additional occurrences add addresses.

	string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", string_join(params.hostnames, ",").c_str()),
	string_format("IP addresses to listen on, comma-separated, or UNIX socket paths ending in .sock (default: %s)", params.hostnames[0].c_str()),

Conversation

erusev commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Sep 10, 2026

Uh oh!

ggml-gh-bot Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR 28690: multiple --host binding

Blocking

Will slow the review

Nits

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

erusev commented Sep 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erusev commented Sep 11, 2026

Uh oh!

ngxson commented Sep 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

erusev commented Sep 10, 2026 •

edited

Loading

ggml-gh-bot Bot commented Sep 10, 2026 •

edited

Loading

Review of PR 28690: multiple `--host` binding