Add multi-session lifecycle support to the v2 runtime by gtong-nv · Pull Request #548 · NVIDIA/flashdreams

gtong-nv · 2026-08-31T19:59:07Z

Summary

Add multi-session lifecycle support while keeping the application, model
weights, and client window alive between sessions.
Add an ImGui prompt UI for T2V applications that can request new sessions
before or after inference.
Keep UI processing active after model completion and across WebRTC
reconnects.
Make IModelLoop the single source of inference lifecycle state.
Suppress the known non-fatal aioice STUN retry race without hiding other
asyncio errors.

fixes [Session Runner] Make IO-Thread/UI-Thread reusable and able to restart (end_old+start_new) a ISession #502

copy-pr-bot · 2026-08-31T19:59:10Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

gtong-nv · 2026-09-01T18:28:31Z

/ok to test cc238c2

greptile-apps · 2026-09-01T18:32:39Z

Greptile Summary

The PR adds multi-session lifecycle support to runtime-v2 while retaining application resources, client windows, and model weights between sessions.

Centralizes inference completion state in IModelLoop.
Adds reusable T2V prompt sessions and continuous ImGui processing.
Preserves WebRTC connections and presentation state across session changes.
Updates lifecycle, metrics, documentation, and integration tests for the new behavior.

Confidence Score: 3/5

The PR is not safe to merge while an unauthenticated valid WebRTC offer can replace the active browser and gain its video and control capabilities.

The previously reported peer-takeover path remains: the offer handler negotiates any valid request, releases the current peer, and installs the requester as the active video and control peer without an authorization boundary.

Files Needing Attention: flashdreams/flashdreams/runtime_v2/serving/webrtc_server.py

Important Files Changed

Filename	Overview
flashdreams/flashdreams/runtime_v2/application_runner.py	Extends application execution from a single session to repeated sessions while retaining application and client-window resources.
flashdreams/flashdreams/runtime_v2/session_runner.py	Reworks session completion, reset, UI processing, and handoff behavior for multi-session execution.
flashdreams/flashdreams/runtime_v2/serving/webrtc_server.py	Keeps signaling and media resources active across sessions and supports peer replacement.
flashdreams/flashdreams/runtime_v2/webrtc_client_window.py	Adapts the WebRTC client window to persistent multi-session lifecycle and event timing.
apps/t2v/t2v/application.py	Loads the shared model during application initialization and accepts optional per-session prompts.
apps/t2v/t2v/session.py	Adds idle prompt sessions, reusable rollout state, and T2V ImGui UI integration.

Sequence Diagram

sequenceDiagram
    participant B as Browser
    participant W as WebRTC Window
    participant R as Application Runner
    participant S as Session Runner
    participant A as Application
    B->>W: Connect and submit input
    W->>R: Persistent client events
    R->>A: Create session
    R->>S: Run session
    S-->>W: Present generated frames
    S-->>R: Model loop finishes
    R->>A: Create replacement session
    R->>S: Run replacement session
    Note over W,A: Client window and application resources remain alive

_{Reviews (8): Last reviewed commit: "Keep T2V CPU tests headless" | Re-trigger Greptile}

greptile-apps · 2026-09-02T00:43:21Z

+            existing_peer = self._peer_connection
+            if existing_peer is not None:
+                # This server has one client slot. A valid new offer is a page
+                # refresh or replacement browser and takes ownership of it.
                try:
-                    self._buffer_browser_message(
-                        message,
+                    await self._release_peer_connection(existing_peer)
+                except BaseException:
+                    await self._release_peer_connection(
+                        peer_connection,
+                        detached_video_track=video_track,
                    )
-                except ValueError as error:
-                    channel.send(json.dumps({"type": "error", "message": str(error)}))
-
-            @channel.on("close")
-            def on_close() -> None:
-                if is_reliable_control:
-                    self._record_client_disconnect()
+                    raise

-        @peer_connection.on("connectionstatechange")
-        async def on_connectionstatechange() -> None:
+            self._media_connected.clear()
+            self._final_video_track_metrics = None
+            self._peer_connection = peer_connection


Valid offers hijack active peers

If the server is bound to an interface reachable by another principal, any valid unauthenticated offer releases the legitimate peer and becomes the active peer, allowing the replacement browser to send keyboard, mouse, touch, reset, close, and new-session input to the running application.

How this was verified: The unauthenticated offer route unconditionally installs a successfully negotiated replacement peer, whose control-channel messages then pass the active-peer guard.

Knowledge Base Used:

Runtime-v2 application execution

Realtime and WebRTC serving

ArielG-NV

Comments on design.

ArielG-NV · 2026-09-02T01:03:17Z

+            loopResult = ui_loop._begin_run(events, generation)
+            if loopResult.stop_requested:
+                stop.set()
+                return
+            if loopResult.new_session_request is not None:
+                next_session_desc = replace(
+                    session_desc,
+                    metadata={
+                        **session_desc.metadata,
+                        **loopResult.new_session_request,
+                    },
+                )
+                stop.set()
+                return
+            if loopResult.step_index is None or not step_requested:
+                return


We can begin a run, and then never finish the run or stop the ui_loop? We should be very careful about introducing cases where we can begin something but never end them.

We should potentially think about even making stop flag setting a type of _finish_run for clarity reasons. A run should likely be forced to have a begin/end for debugging/code-readability/code-extensibility reasons.

We can begin a run, and then never finish the run or stop the ui_loop

For serving, this is actually a valid use case. If we want to stop, we just kill the server/app.
For debugging purpose, yea, it's better to have a flag to stop the UI run even for the unbounded interactive case.
But how should we design the flag? something like --timeout 60 for 1-min run and stop the app entirely?

_finish_run should really just be in a finally: block I suppose... and if this is not possible then something is wrong (begin at entry, finish at end)

oh I thought you were referring "launching the app, and let the app run forever".
yea, if what we want is matching _begin_run and _finish_run, this makes sense. I will update that

Moved _finish_run to finally: block

gtong-nv · 2026-09-02T22:55:26Z

/ok to test 5a0933b

ArielG-NV

Minor suggestions; PR looks very good now.

gtong-nv · 2026-09-03T01:55:32Z

/ok to test 7750f1c

ArielG-NV

PR looks good now, just need to fix failing tests.

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

aioice can run a retry callback after its transaction future has already completed, which produces a non-fatal InvalidStateError. Filter only that exact callback failure on the dedicated WebRTC event loop and delegate every other exception to asyncio. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Move prompt-driven session replacement into the application UI and keep the runtime UI loop responsive before and after model inference. Model completion no longer implicitly terminates a session; lifecycle events, UI completion, explicit limits, and failures remain terminal. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Let each model loop track its own inference lifecycle and have UI loops query that single source of truth. Initialize the state through the common loop registration path and remove the redundant session-runner wrapper. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Parse and fully negotiate replacement offers before releasing the active peer. Serialize offer swaps and clean up failed candidates without disconnecting the connected browser. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Let UI loops request a fully resolved SessionDesc and have run_session forward it unchanged. This allows replacement sessions to modify any description field instead of only merging metadata. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Co-authored-by: ArielG-NV <159081215+ArielG-NV@users.noreply.github.com> Signed-off-by: Gangzheng Tong <tonggangzheng@gmail.com>

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

ArielG-NV · 2026-09-03T17:48:39Z

/ok to test 5d48dcc

gtong-nv force-pushed the dev/gtong/session-management branch from f5348d0 to 07b9786 Compare August 31, 2026 21:29

gtong-nv commented Sep 1, 2026

View reviewed changes

Comment thread flashdreams/flashdreams/api_v2/loop.py

gtong-nv marked this pull request as ready for review September 1, 2026 18:28

greptile-apps Bot reviewed Sep 1, 2026

View reviewed changes

Comment thread flashdreams/flashdreams/runtime_v2/serving/webrtc_server.py Outdated

greptile-apps Bot reviewed Sep 2, 2026

View reviewed changes

ArielG-NV reviewed Sep 2, 2026

View reviewed changes

Comment thread flashdreams/flashdreams/runtime_v2/session_runner.py Outdated

Comment thread flashdreams/flashdreams/api_v2/loop.py Outdated

ArielG-NV approved these changes Sep 3, 2026

View reviewed changes

gtong-nv and others added 14 commits September 3, 2026 16:41

Add multi-session lifecycle support to the v2 runtime

50c36a4

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Preserve active WebRTC peers on invalid offers

bf1f650

Parse and fully negotiate replacement offers before releasing the active peer. Serialize offer swaps and clean up failed candidates without disconnecting the connected browser. Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Update v2 window and loop lifecycle

3076c78

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Fix v2 loop registration and pacing tests

d1b7dec

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Add application timeout to v2 CLI

4846340

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Pair v2 loop run lifecycle

5881b24

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Update flashdreams/flashdreams/runtime_v2/session_runner.py

68a4a27

Co-authored-by: ArielG-NV <159081215+ArielG-NV@users.noreply.github.com> Signed-off-by: Gangzheng Tong <tonggangzheng@gmail.com>

Remove WebRTC new-session input event

0975bfd

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

End default UI after final model frame

e4b3467

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

Keep T2V CPU tests headless

5d48dcc

Signed-off-by: Gangzheng Tong <gtong@nvidia.com>

gtong-nv force-pushed the dev/gtong/session-management branch from a0c9e56 to 5d48dcc Compare September 3, 2026 16:59

gtong-nv added this pull request to the merge queue Sep 3, 2026

Merged via the queue into main with commit 41846b7 Sep 3, 2026
9 checks passed

gtong-nv deleted the dev/gtong/session-management branch September 3, 2026 18:38

This was referenced Sep 3, 2026

Collect model input after cadence pacing #573

Open

Add correlated V2 runtime latency profiles #574

Open

Conversation

gtong-nv commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

copy-pr-bot Bot commented Aug 31, 2026

Uh oh!

Uh oh!

gtong-nv commented Sep 1, 2026

Uh oh!

greptile-apps Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

greptile-apps Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

ArielG-NV left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ArielG-NV Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

gtong-nv Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArielG-NV Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gtong-nv Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

gtong-nv Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gtong-nv commented Sep 2, 2026

Uh oh!

ArielG-NV left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gtong-nv commented Sep 3, 2026

Uh oh!

ArielG-NV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArielG-NV commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gtong-nv commented Aug 31, 2026 •

edited

Loading

greptile-apps Bot commented Sep 1, 2026 •

edited

Loading

gtong-nv Sep 2, 2026 •

edited

Loading

ArielG-NV Sep 2, 2026 •

edited

Loading

ArielG-NV left a comment •

edited

Loading