Repository navigation
server: add --connect, allow p2p access to server outside LAN - #28277
Conversation
github-actions
Bot
added
documentation
There was a problem hiding this comment.
this is similar to pulling pre-built llama-ui, so CC'ing @aldehir if you want to take a look
|
Pushed a small fix commit on the branch. I built llama-connect locally and tested this on Linux: the tunnel works end to end through the public trackers, in router mode and with a model loaded. Teardown is clean, the child is gone about 200 ms after SIGTERM and it also exits on its own when I SIGKILL the server, so the stdin pipe trick does exactly what it promises. The API key is not bypassed either, a tunneled request without a key still gets a 401. Two things I fixed. First, --connect-code without --connect was still served in props, so the Web UI would show a remote access code with no tunnel behind it. It is now cleared, with a warning at startup. Second, the comment above terminate() says it closes the child stdout, which it does not: the log thread only leaves fgets() once every writer of that pipe is gone. With a fake llama-connect that leaves a child behind, the server stays stuck in stop() indefinitely. That cannot happen today since llama-connect never spawns anything, so I only rewrote the comment to state the constraint. I also reverted an unrelated README change that flipped the --reasoning-preserve default. What I left alone: making the log reader poll instead of blocking on fgets would remove that constraint for good, roughly what the MCP stdio transport already does, but that is your call on how to factor it. And when the child dies on its own we keep a zombie until the server exits, while props keeps advertising a code that no longer works, so the Web UI has no way to know the tunnel is down. Not about this PR, but it affects both llama-connect and the Web UI. Once a browser connects, the host stays out of the tracker swarm until its next announce. A second client 30 seconds later gets in fine, one right after gets no answer at all. On the UI side the offer is announced once and then we wait 30 seconds, which is exactly the host announce interval, so a page reload lands in that window about half the time. Re-announcing right after sending the answer would fix the cause on the host, and re-sending the offer periodically instead of once would make the client survive a lost announce, which public trackers do anyway. |
|
CC @ggerganov if you can have a quick look on this (mostly the cmakelists part) A bit hard to fully test this feature rn as we are still missing the web ui piece, but that will be ready soon just after this PR.
Not sure about this but I think it can be fixed on llama-connect source code or web ui, or both? |
|
Rebasing this one |
--connect-code on its own had no effect other than being served in /props, so the Web UI showed a remote access code that no tunnel was serving. Clear it when --connect is off, and say so once at startup. The teardown comment claimed that terminate() closes the child stdout. It does not: the log thread leaves fgets() only once every writer of that pipe is gone, so a descendant of llama-connect would block the join forever. Describe the real contract instead. Also restore the --reasoning-preserve default in the generated docs.
a2ec6b0 to
33e296a
Compare
|
Rebased on master: adapted to the multi address binding from #28690 (hostname -> hostnames, listening_addresses, ctx_http.join()) and dropped the :8080 notice now that the default port is 9931. Nothing else changed in the connect code. |
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
|
Two small things I noticed while rebasing, not blocking:
|
Pass ctx_http.port so --port 0 and the GCP override work, pick the dial host from --host (wildcards map to loopback, IPv6 is bracketed) and refuse --connect when only unix sockets are bound.
|
Fixed in 136b832. llama-connect now gets the port the server actually bound, so --port 0 and the GCP override work. The host it dials comes from --host: 0.0.0.0 maps to 127.0.0.1, :: to [::1], a specific address is used as is, and a server listening only on unix sockets now refuses --connect at startup instead of running a dead tunnel. I checked the llama-connect side for IPv6: it builds the upstream as http://{host}:{port} with no brackets, so the server passes IPv6 literals already bracketed. Its startup /health check only logs a warning on failure, which is why a wrong port or host used to fail silently. Tested with a local llama-connect build: --port 0, 0.0.0.0, ::, ::1, a LAN address and a mixed unix socket + TCP setup all reach "llama-server is up", and no child is left behind on exit. |
|
hm, CI for Vulkan is failing for some reason, @ServeurpersoCom please take a look |
Not this PR: gpu-vulkan-nvidia-cm2 crashes in test-llama-context with heap corruption at exit, and the same job is flaky on master with the same test (7 of the last 12 master runs red since yesterday), so a re-run should do it; it needs a separate CI fix. |
|
merging this then |
|
we also need to update brew build: pre-built binary is currently pulled from internet by cmake, it need to be downloaded before cmake runs |
I will take over once i come back directly to implementing this feature |
…rg#28277) * init * wip * wire it up * make it clear that it's experimental * add props.connect_code * nits * fix windows build * server: do not advertise a connect code without the tunnel --connect-code on its own had no effect other than being served in /props, so the Web UI showed a remote access code that no tunnel was serving. Clear it when --connect is off, and say so once at startup. The teardown comment claimed that terminate() closes the child stdout. It does not: the log thread leaves fgets() only once every writer of that pipe is gone, so a descendant of llama-connect would block the join forever. Describe the real contract instead. Also restore the --reasoning-preserve default in the generated docs. * Update tools/connect/CMakeLists.txt Co-authored-by: Adrien Gallouët <angt@huggingface.co> * server: dial llama-connect on the bound port and a reachable host Pass ctx_http.port so --port 0 and the GCP override work, pick the dial host from --host (wildcards map to loopback, IPv6 is bracketed) and refuse --connect when only unix sockets are bound. --------- Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co>
Overview
Add
llama-server --connectmode that allow opening a p2p connect between web UI and llama-serverRelated web UI PR: #24577
This requires pulling pre-built binary from https://github.com/ggml-org/llama-connect as the project is based on rust. While we can implement it in C++, that will end up adding too many 3rd-party dependencies, making the build large and not very portable. The rust build is statically linked, should be more portable. This decision keeps the maintenance cost low in the long run.
Protocol is built with privacy in mind:
TODO:
macos-x64, currently we don't have a build for it because GH action doesn't seem to have the capacityRequirements