Repository navigation
Conversation
Fair point on caution, but to clarify: this doesn't expose llama-server to the WAN. The server stays bound to localhost, the host browser just proxies same-origin requests over an encrypted p2p data channel, so nothing is actually listening on a public interface. That warning is about binding to a public address, which this setup doesn't do. wireguard/tailscale are great but they need per-device VPN setup, the whole point here is zero-config in the browser for non-technical users. |
|
@ngxson great PoC :D will take a closer look at this in near future :) |
|
I'm going to rebase it and test it |
41a58ff to
d31babc
Compare
|
@allozaur @ServeurpersoCom with #28277 in a working state now, let's try shipping this feature somewhere this month? one important thing to decide is whether to push a pre-built version of llama-ui to |
b5be349 to
254f968
Compare
UI side rebased |
The sidecar binary runs next to llama-server and covers the headless case, so the browser no longer needs to act as a host. Only the client side remains, the wire protocol is unchanged.
A local flag tracked the same thing as the tunnel status, so leaving an attempt in progress restored the form while the pending call still held the flag, leaving the code field disabled. The settings screen had the same split across two flags.
The saved session becomes an array of entries discriminated by type, so later work can add servers and other kinds of connection without changing the stored shape. Entries hold the share code as the user pastes it, leaving the room and pass split as a tunnel detail.
7c30fcd to
a674df1
Compare
|
Rebased |
A remote llama.cpp server is one the UI reaches over a WebRTC data channel instead of a base URL. It stays a provider: it is listed, named, enabled and disabled, and its models appear in the selector like everyone else's. What changes is where a provider's requests are addressed. A remote provider has no URL of its own, so it is given a synthetic origin whose host carries its id (`https://<id>.remote.llamacpp.invalid`, the suffix being reserved so the host can never resolve). Every service already builds its URLs through `apiUrl`, so nothing in the services had to learn about tunnels: `getBackendBaseUrl` hands back the synthetic origin, and a fetch wrapper installed in `hydrateStores` routes a request by that host instead of by the page origin. Routing on the address is what makes this safe to run next to other providers. The first remote-access build tunneled everything aimed at the page origin, so one tunnel was the most it could hold, and a remote provider with a URL of its own would have slipped past it. Now a request aimed at a tunnel origin with no live tunnel is refused out loud, and because the origin cannot resolve, a call site that ever bypasses the resolver dies in the browser rather than reaching the server that happens to serve this UI. A request queued while the handshake ran is refused the same way if the channel died underneath it. Tunnels open lazily: only the provider the app opens with is handshaked during startup, the rest when one of their models is picked or Connect is hit on their card. A provider that is disabled or removed loses its data channel, and its code - which is a credential - is dropped with it. Three things this forces into the open, all of them pinned to the built-in provider back when no second provider could exist: - `validateApiKey` probed the active provider with the key of the local one, so a remote provider answering 401 raised the API key splash over a credential belonging to another machine. - `streamTool` ran tools on the local server while the model asking for one could be anywhere. A tool now runs where its tokens are generated. - settings export scrubbed `apiKey` and MCP headers but exported the provider list whole, keys included. Remote providers now leave the export entirely: without their code they name nothing, and with it they are a key. The tunnel itself comes from ggml-org#24577 (ngxson), with the base-path stripping dropped: a request already carries the path the host's server serves. Co-Authored-By: ngxson <ngxson@gmail.com>
Overview
This is a PoC that demonstrates how to use llama.cpp remotely on a mobile device via p2p webrtc connection. It is NOT intended to be a production-ready, but can be useful for discussion.
My main goals are:
To achieve that, implementation is as follow:
fetch()calls are intercepted and are tunneled via webrtc connectionUser flow:
Caveats:
Demo
20260613_162415.mp4
Points for discussion
ui.llama.app(?)Requirements