[TRTLLM-16460][feat] agent-flow: trtllm_modeling_v2 workflow for gated ModelingV2 onboarding and perf tuning

AGENTS.md

@@ -193,6 +193,21 @@ For a full list of up-to-date bot commands, post `/bot help` as a PR comment and
 
 - Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `"debug:_torch,runtime;info:serve"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details.
 
+**Environment failures that read as model failures.** Each of these presents as a broken
+model or a broken build, and none of them is. Check them before debugging modeling code:
+
+| symptom | cause | fix |
+|---|---|---|
+| `MPI_ERR_SPAWN: could not spawn` | the container runtime forbids `MPI_Comm_spawn`, which the default LLM API path uses to start workers (normal under pyxis/enroot) | `TLLM_WORKER_USE_SINGLE_PROCESS=1` |
+| `Impossible to fit in any sequence in kvCache` | the engine inherited the checkpoint's `max_position_embeddings` (262144 is common) and sized the cache for it | pass `--max_seq_len` / `--max_batch_size` / `--kv_cache_fraction` explicitly |
+| `NVRTC ... could not open source file "cuda.h"` | usually the venv's `bin` missing from `PATH`; separately, the generation FMHA kernel takes the NVRTC path unless the KV dtype is `E2m1` (grep `options.mDtypeKv != tg::Dtype::E2m1`) | check `PATH` first; `--kv_cache_dtype fp8` takes the non-NVRTC branch |
+| the engine hangs at startup and never returns | a wrong launch substrate hangs rather than raises — typically an MPI connect-back on an interface that does not carry it | probe with a deadline rather than waiting |
+| the import resolves to a different checkout | `/root/.local` is shared across container runs on some clusters | `PYTHONNOUSERSITE=1` at runtime — but **not** during the native build, which needs the user site for `cutlass_library` |
+
+A longer list, with the wrong diagnoses recorded alongside the right ones, is in
+[`agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md`](agent-flow/agent_flow/workflows/trtllm_modeling_v2/TROUBLESHOOTING.md).
+It is keyed on verbatim error text, so search it for the string you saw.
+
 ## Key Documentation
 
 | Topic | Path |

agent-flow/README.md

@@ -297,4 +297,5 @@ block, before `run`: it writes each role's composed system prompt verbatim to
 handed in, since that is what the agents are built from (`AgentTeamWorkflow`
 appends its MCP-tool protocol to the bundle it is given). The `perf-analyze`
 and `perf-optimize` CLIs use it to snapshot `<workspace>/prompts/` at every
-launch.
+launch; `trtllm-modeling-v2` does the same from preflight, once the launch
+environment its prompts carry is resolved.

agent-flow/agent_flow/runtime/__init__.py

@@ -0,0 +1,70 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Execution substrate shared by every workflow.
+
+Two questions live here, and neither of them is a workflow's business:
+
+* **how does one agent turn get delivered and answered** — in this process
+  through an ``AgentLayer``, or through a filesystem mailbox a separate
+  authenticated session drains;
+* **where and how does one command run** — a local subprocess, an ``srun``
+  step, or a file dropped into a spool a long-lived allocation watches.
+
+A workflow states *what* it needs run and *what* it needs proven. It should
+not carry a branch per cluster protocol, and it should not have to change
+when a credential store turns read-only. Everything in this package is
+therefore free of workflow vocabulary: nothing here knows what a catalog
+entry, a smith or a gate is.
+
+A third concern, older than the other two, already lived at this import
+path: running async work on a background loop for a synchronous caller. It
+is the same kind of thing — substrate a workflow uses without describing —
+so it stayed, as :mod:`~agent_flow.runtime.portal`, and
+``from agent_flow.runtime import PortalRunner`` still resolves.
+"""
+
+from .commands import (
+    CommandExecutor,
+    CommandSpec,
+    DelegatedCommandExecutor,
+    ExecutionCapabilities,
+    ExecutionRecord,
+    LocalCommandExecutor,
+    SlurmStepExecutor,
+    SpoolCommandExecutor,
+    SubmissionRenderer,
+)
+from .portal import PortalRunner, build_request
+from .turns import (
+    DEFAULT_TURN_TIMEOUT_S,
+    FileDispatchTurnExecutor,
+    InProcessTurnExecutor,
+    TurnExecutionError,
+    TurnExecutor,
+    TurnOutcome,
+    TurnRequest,
+    TurnTimeout,
+)
+
+__all__ = [
+    "CommandExecutor",
+    "CommandSpec",
+    "DEFAULT_TURN_TIMEOUT_S",
+    "DelegatedCommandExecutor",
+    "SubmissionRenderer",
+    "ExecutionCapabilities",
+    "ExecutionRecord",
+    "FileDispatchTurnExecutor",
+    "InProcessTurnExecutor",
+    "LocalCommandExecutor",
+    "PortalRunner",
+    "SlurmStepExecutor",
+    "SpoolCommandExecutor",
+    "TurnExecutionError",
+    "TurnExecutor",
+    "TurnOutcome",
+    "TurnRequest",
+    "TurnTimeout",
+    "build_request",
+]

agent-flow/agent_flow/runtime/commands/__init__.py

@@ -0,0 +1,24 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Command execution: one interface, one per place a command can land."""
+
+from .base import CommandExecutor, CommandSpec, ExecutionCapabilities, ExecutionRecord
+from .delegated import DelegatedCommandExecutor, SubmissionRenderer
+from .local import NOT_FOUND_RETURNCODE, TIMEOUT_RETURNCODE, LocalCommandExecutor
+from .slurm import SlurmStepExecutor
+from .spool import SpoolCommandExecutor
+
+__all__ = [
+    "CommandExecutor",
+    "CommandSpec",
+    "DelegatedCommandExecutor",
+    "ExecutionCapabilities",
+    "ExecutionRecord",
+    "LocalCommandExecutor",
+    "NOT_FOUND_RETURNCODE",
+    "SlurmStepExecutor",
+    "SubmissionRenderer",
+    "SpoolCommandExecutor",
+    "TIMEOUT_RETURNCODE",
+]

agent-flow/agent_flow/runtime/commands/base.py

@@ -0,0 +1,140 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""One command, and where it ran.
+
+A workflow that measures something needs to say *what to run* and *how long
+it may take*. Where that lands — this host, an ``srun`` step, a file in a
+spool a long-lived allocation drains — is a property of the deployment, and
+every time it leaked into the workflow it arrived as a new branch beside a
+stage transition.
+
+So: the caller builds a :class:`CommandSpec`, an executor returns an
+:class:`ExecutionRecord`, and the record carries enough provenance that the
+verdict drawn from it can be defended later without knowing which executor
+produced it.
+
+**A timeout is not an error here.** ``timed_out`` is a separate field from
+the exit code because the two need different handling — a gate that timed out
+has said nothing about the thing it was gating, while a gate that exited
+non-zero has.
+"""
+
+from __future__ import annotations
+
+import abc
+import shlex
+from dataclasses import dataclass, field
+from typing import ClassVar, Mapping, Sequence
+
+
+@dataclass(frozen=True)
+class ExecutionCapabilities:
+    """What an executor can actually run, in machine-readable form.
+
+    Stated rather than inferred, so a caller can refuse an impossible
+    request before paying for it instead of discovering the limit from a
+    failure three hours in.
+    """
+
+    gpu: bool = False
+    distributed: bool = False
+    #: True when the executor runs inside an allocation acquired once and
+    #: reused, rather than queueing per command. That is the difference
+    #: between paying a scheduler wait (and a container import) per gate and
+    #: paying it once for the run.
+    persistent_allocation: bool = False
+    max_world_size: int | None = None
+    architectures: tuple[str, ...] = ()
+
+
+@dataclass(frozen=True)
+class CommandSpec:
+    """What to run, and the bounds it runs under.
+
+    ``argv`` is either a **string**, which is a shell line and is run as
+    one, or a **sequence**, which is executed directly with no shell between.
+    Both forms are real and neither is a convenience: a resolved launch
+    prefix (environment assignments, or an ``srun`` line) *is* shell syntax
+    and re-parsing it into an argv would mean reimplementing the shell, while
+    a probe like ``nvidia-smi --query-gpu=...`` has no reason to be exposed
+    to word splitting at all.
+    """
+
+    argv: str | Sequence[str]
+    env: Mapping[str, str] | None = None
+    timeout_s: int = 600
+    #: A short name for this command in logs and artifacts. Not an id.
+    label: str = ""
+    #: How many ranks the command expects. Carried so an executor can refuse
+    #: a request its allocation cannot satisfy; never used to rewrite the
+    #: command, which is the caller's business.
+    world_size: int = 1
+    gpu_arch: str | None = None
+    #: Where to run it, when the executor supports a working directory.
+    cwd: str | None = None
+
+    @property
+    def rendered(self) -> str:
+        """``argv`` as a single line, for logs and for shell execution."""
+        if isinstance(self.argv, str):
+            return self.argv
+        return " ".join(shlex.quote(str(part)) for part in self.argv)
+
+    @property
+    def is_shell_line(self) -> bool:
+        return isinstance(self.argv, str)
+
+
+@dataclass(frozen=True)
+class ExecutionRecord:
+    """What happened, with the provenance a verdict rests on."""
+
+    command: str
+    returncode: int
+    stdout: str = ""
+    stderr: str = ""
+    timed_out: bool = False
+    duration_s: float = 0.0
+    #: Which executor ran it — ``local``, ``slurm``, ``spool``.
+    executor: str = ""
+    label: str = ""
+    #: Where the full output lives, when the executor wrote one. The tail a
+    #: record carries is for reading; this is for auditing.
+    log_path: str | None = None
+    started_at: float = 0.0
+    artifacts: tuple[str, ...] = field(default_factory=tuple)
+
+    @property
+    def ok(self) -> bool:
+        return self.returncode == 0 and not self.timed_out
+
+    @property
+    def combined(self) -> str:
+        return self.stdout + self.stderr
+
+    def tail(self, lines: int = 40) -> str:
+        return "\n".join(self.combined.splitlines()[-lines:])
+
+
+class CommandExecutor(abc.ABC):
+    """Run one command somewhere and report what happened."""
+
+    #: Recorded in :attr:`ExecutionRecord.executor`.
+    name: ClassVar[str] = "command-executor"
+
+    #: Overridden per executor. A class attribute rather than a property so
+    #: a caller can ask without constructing one.
+    capabilities: ExecutionCapabilities = ExecutionCapabilities()
+
+    @abc.abstractmethod
+    def execute(self, spec: CommandSpec) -> ExecutionRecord:
+        """Run ``spec`` to completion or to its deadline."""
+
+
+__all__ = [
+    "CommandExecutor",
+    "CommandSpec",
+    "ExecutionCapabilities",
+    "ExecutionRecord",
+]