Plans of continued development?

Hi,

Out of curiosity, are there any plans to continue developing these nodes? I don't have any expectations, neither am I implying anything. Just wondering, since the last development activity was 2 years ago, and there's multiple unanswered issues, and of course a host of new models and architectures.

The real question should be, to whom do we hand off this project? Or perhaps at this point, which of the forks do we agree to redirect the visitors of this abandoned resource? How do we convey this reliably?

If my quick observation is correct... wait what does comfyanonymous imply? This resource is in hands of the ComfyUI team? Well then I'd say we poke them to answer these questions. PRs aren't getting merged...

Quite a few questions there, and, honestly, my initial question was a much more generic one. I was just cleaning up my Comfy custom nodes and wasn't sure if this node was worth keeping. For now I ended up disabling it.

You seem to have multiple questions, more complex and layered, and that are more suited for a discussion. Unfortunately at the moment it doesn't really feel that it's a discussion anyone is interested in having.

Be interesting to see where this goes - because as of the last ComfyUI update, I cannot get this working any more.

After converting Models, this was significantly faster at rendering.

I've made a VAE TensorRT implementation. Or rather, I've managed to let Claude manage to do so. The goal was to dip my toes in how complicated TensorRT is. We started with SDXL VAE. To my surprise in a XY Plot workflow this can squeeze out another 30% roughly on the workflow time compared to only SDXL TensorRT. So thar was a success.

Then the steep hill came. For some reason VAE encode is much more complicated than decode in Wan/Temporal contexts. After too much time we gave up on the Wan VAE encode implementation and only managed the VAE decode implementation.

Now that was solely for the VAE. So I reckon another Mount Everest is waiting in the actusl sampler implementation. That's a very rough guess though. I haven't paid enough attention to Claude's rambles to deeply understand what TensorRT is and where is can and cannot be applied. I wildly guess it depends on the domain and range of the operations. Lemme ask Gemini.

Does the ability to implement tensorrt on an ai model depend kn the domain and or range of the models architecture

Yes. When applying an ahead-of-time (AOT) compiler like TensorRT to an AI model, the mathematical concepts of "domain" and "range" translate directly to input tensor specifications and activation/output boundaries. TensorRT's ability to optimize and compile the model is strictly gated by these constraints.

I've always been good at catching information while barely paying attention. I still got it.

In short, if anyone is interested in SDXL VAE implementation and or Wan VAE Decode implementation lemme know and I'll take a minute to publish a repo. I'd say the SDXL VAE is great for XY Plot and otherwise very niche. Likewise the Wan VAE step, eh, it's a small performance bump.
But do not expect a high quality repo. It was for proof of concept and violates all ComfyUI LiteGraph principles - actually almost every comfy nodes repo does that.

I sometimes wonder if the comfyui abstraction is helping or not at all. I suppose the answer is to develop in Comfy and port to raw code in prod if you need performance.

Update

I loaded up the claude chat where I left off. Turns out I had already started on "DiT TRT test - ONNX CPU export blocker". I remember now that I asked a Claude agent to do some ground work while the VAE agent is churning (and failing). I did that nearing a Claude Quota Reset. Always a good opportunity to let an orchestrator agent spawn parallel Opus workers to spend those tokens and see what comes out of it.

Chat: "DiT TRT test - ONNX CPU export blocker"

It's been a few weeks. What is the state? Am I right the initial WAN VAE decode was slower and we are trying something now? And is it true the WAN VAE encode is not viable? Why?

● Let me refresh my memory from the project files.

Here's the state:

WAN VAE TRT — abandoned. Both decode and encode are dead ends:

Decode: The split architecture puts TRT on the cheap latent-resolution layers, while the expensive pixel-resolution layers stay in PyTorch (because they need frame-by-frame temporal caching via CausalConv3d, which TRT can't do). Result: 2x slower than vanilla PyTorch. On top of that, fp16 GroupNorm produces dark, crushed output — a precision disaster.
Encode: Same fundamental issue — temporal compression requires CausalConv3d caching. TRT full-pass mode preserves T instead of compressing it (33 frames → 33 instead of → 9). Since video encode always has multiple frames, there's no useful case. All WAN encode just delegates to the original PyTorch VAE.

Current focus: WAN DiT TRT — accelerating the diffusion backbone (14B model, 20–50 denoising steps). That's where TRT actually matters. We got through ONNX export + ORT validation, then this session was adding TRT engine build + inference test steps (Steps 4–5). The test hit a blocker: the dequantized FP8→fp16 model (~26 GB) gets pushed to CPU for ONNX tracing, but Conv3d has no CPU kernel (aten::cudnn_convolution). We were freeing disk space to see if keeping it on GPU was feasible, but 16 GB VRAM vs 26 GB model is the real constraint.

Okay. But as for the WAN VAEs, especially the decode — is it determined there is no speed to be gained using any TensorRT impl?

Not fully determined — there's one approach the roadblocks doc mentions as "not attempted" that could work:

Explicit cache I/O. The reason TRT can't do the cached decoder is that CausalConv3d maintains state between frames. But you could make those cache tensors explicit inputs and outputs of the TRT engine — similar to how TRT-LLM handles KV-cache in LLMs. Each frame call would be: (frame, cache_in) → (output, cache_out). TRT handles the heavy compute, cache state lives in regular CUDA tensors between calls.

This would let TRT run the full decoder per-frame, not just the cheap latent-res layers — addressing the "TRT accelerates the wrong part" problem.

Per-layer precision (GroupNorm in fp32, Conv3d in fp16) was also not attempted. TRT supports this via layer.precision constraints. Would fix the dark output.

Combined, those two could potentially give a real speedup. But:

The ONNX export would need significant rework — every CausalConv3d's cache becomes an explicit tensor pair (input + output), and there are many of them
VAE decode is called once per generation. The DiT runs 20–50 steps. Even a 3x VAE speedup barely moves the needle on total generation time
It's a lot of engineering for marginal end-to-end impact

So: not impossible, but the ROI is poor compared to DiT TRT, which is why we pivoted there.

So

I guess I'll let some agent poke at it on the side while I'm working. With some luck Wan DiT TRT Hack Magic is pretty straightforward and it'll be done swiftly. (Though likely sub-optimal initially)

This sounds interesting tho, and kudos for doing testing of this 👍
I am a complete n00b to this, but i have been using the RIFE Tensorrt nodes + Tensorrt Upscaler nodes, and with some tweaks managed to make some updates for adding x2 and x1 support to various upscale models.
I prefer just using a x1-SkinDIFF model for some final polishing in a 30-60 second wan video over 2x or 4x upscaling, because i find the result being overly "smooth".

So, a TensorRT for VAE encode/decode - especially for wan - could very well be something interesting. I am all for a bit of tinkering and testing, but i come up very short when it comes to python and torch-whatever 😏

So, a TensorRT for VAE encode/decode - especially for wan - could very well be something interesting. I am all for a bit of tinkering and testing, but i come up very short when it comes to python and torch-whatever 😏

Yeah but my Claude Opus and my findings are that the VAE for Wan is not significantly TensorRT-able, unfortunately.

Claude has predicted that the inference models, in contrast, are definitely optimizable using this tech.

I will try and make progress when I have time for it. Don't rely on me though. This is low-prio. It's just for fun. I don't need Wan to be faster for my personal use. I only wish to have some prod project where inference_time=money.

https://github.com/NubeBuster/ComfyUI_TensorRT

I have a patching system where I can write custom patches to quickly solve bugs and implement missing features on repos without worrying about quality, leaking my creds on tha street, and all the other shenanigans.

Having 4 sequentially dependent patches lead to too much LLM work continuing onwards. So I reckoned, tested, useful, sharable:

Here follows documentation in a conversational format. The base markdown is the information the LLM stated, the quotes are my notes

ComfyUI TensorRT Fork — Patch Summary

1. ONNX Export Compatibility

Fixes ONNX export failure on modern ComfyUI. Disables comfy_cast_weights during tracing (its .view(dtype=...) produces unsupported ops), handles ModelPatcherDynamic by skipping force_patch_weights, and ensures all weights are on GPU before export.

2. Step Size Reduction

Reduces height/width input step from 64 to 16. Allows finer-grained resolution control for TRT engine profile shapes without being locked to 64px increments.

3. Lazy Engine Load + VRAM Reporting

Defers TRT engine deserialization to first use and reports true VRAM cost (weights + context scratch) to ComfyUI's memory manager. Previously only context memory (~50 MB) was reported, causing ComfyUI to freely evict and reload 5 GB engines between XY plot iterations.

4. LoRA Refit (UNTESTED)

Adds TensorRTRefitLoader node and enable_refit builder flag. Build an engine once with refit enabled, then swap LoRA weights in seconds via trt.Refitter instead of rebuilding from scratch (minutes). Extracts patched weights from source model, maps PyTorch keys to TRT names, and refits in-place.

UNTESTED, CLAUDE SPAT IT OUT 5m AGO, I WILL NOW TRY AND UNDERSTAND WHAT IT CLANKED INTO MY FILESYSTEM AND REPO. If we're lucky, this allows for quick rebuild of in-memory TRT engines when adjusting LoRA strengths.

────────────────────────────────────────────────────────────────────────────────────────
❯  So in a oneliner: what is possible with this commit
   And in a very concise step by step, how can one take advantage?
────────────────────────────────────────────────────────────────────────────────────────

TensorRT LoRA Refit

Swap LoRA weights (any combination, any strength) into a pre-built TRT engine in seconds instead of rebuilding from scratch.

Note: Refit is in-memory only — the engine file on disk keeps its original base weights. Each execution re-refits (still seconds, not minutes), but nothing is persisted to disk.

Setup (one-time)

Load your base checkpoint (no LoRA)
Connect to Dynamic/Static TRT Conversion with enable_refit: True
Build — this takes minutes but only happens once

Usage (per-LoRA , fast)

Load checkpoint → Load LoRA(s) → MODEL output
Connect MODEL to TensorRT Refit Loader (source_model)
Select your engine file and model type
Run — refit takes seconds, generates at TRT speed
Change LoRA/strength/stack → rerun → seconds again

~~So, LoRA application lives until server restart.~~
Correction: engine with LoRA lives until VRAM eviction. It's potentially feasible to transfer it to RAM for temporary persistence. Planned™
Update: upon further interrogation, since these LoRA applied engines can be unloaded to RAM surely they can also be stored on disk? Yes. Elaboration on Planned™ functionality: buttons and stuff for user do it.
I'll see when I get the VAE implementation added to my fork.

────────────────────────────────────────────────────────────────────────────────────────
❯   is it the case that the refit model is a mere intermediary,
    and the lora result engines are basically the same as the normally built lora engines?
────────────────────────────────────────────────────────────────────────────────────────

Yes, exactly. The refitted engine is byte-for-byte functionally identical to one you'd get by building from scratch with the LoRA already applied to the checkpoint. The Refitter API just pokes new weight values into the same slots — the engine architecture, optimizations, and layer fusion are all preserved from the original build.

The refit is just a fast path to the same result.

────────────────────────────────────────────────────────────────────────────────────────
❯  what is the overhead for enabling refit on the base model?
   could I run the model without refitting too?
────────────────────────────────────────────────────────────────────────────────────────

Overhead: Engines built with enable_refit=True are slightly larger and potentially slightly slower — TensorRT can't fuse/fold certain weight operations when it needs to keep them addressable for refitting. In practice the difference is small (single-digit percent), but it's nonzero.

Running without refitting: Yes, the regular TensorRT Loader loads and runs it just fine — it doesn't care whether the engine has refit enabled or not. It just runs the weights that were baked in at build time.

So you could build one refit-enabled engine from your base checkpoint and use it both ways: directly via TensorRT Loader (no LoRA, base weights) or via TensorRT Refit Loader (with LoRA). You don't need separate engines.

af014d6 - TRT-accelerated VAE encode and decode for AutoencoderKL models.

LoRA Refit

SDXL Refit (apply LoRAs to a TRT engine in ~30s not ~5-10m) implemented and working for me.

Implemented, tested, fixed bug, repeat 5 times? Now it works mostly. I've fully focused the debugging on having it work for XY Plotting. In XY plot you don't want to unload the engine between each iteration, or the TRT VAE model..? In short, as far as I know and have tested: - Executing an XY Plot (from EasyUse) would cause model load/unload overhead in my scenario (providing a TRT VAE + Unet REFIT model), the comfyui API allegedly doesn't properly support this niche case where we do not want to unload the TRT VAE nor the Unit model between XY Plot inference iterations on memory pressure, but specifically only during the XY Plot Execution: behave normally for other model eviction events, without modifying the external custom nodes like EasyUse.

SDXL VAE TRT

Benchmark report I mentioned earlier in this issue benchmark was found to be invalid, effectiveness TBD.

I ran one test on a given XY Plot SDXL T2I workflow on fresh ComfyUI restart assuring the refitted model cache on disk is present.
Model: BigLove_Pony3 with a bunch or LoRAs applied that I left unchanged between test runs.
XY-Plot:

#	X-Sampler	X-Scheduler	Y-Seeds
1	dpmpp_2m_sde_gpu	beta	8
2	dpmpp_2m	beta	8
3	euler	beta	8
4	euler_ancestral	karras	8

Total: 32 XY-Plot Iterations (i.e. Images)

Results:

Prompt executed in 335.11 seconds
Prompt executed in 334.05 seconds

Conclusion:

... ah wait dynamic vram - Mar 5th (moreover, new, maybe my VAE TRT Implementation became obsolete, and I haven't failed to properly test it in the past)

Lemme set --disable-dynamic-vram...
VAE No-TRT: 347.61 seconds
VAE TRT: 377.42 seconds

After some hard self-reflection I retraced my steps back to the workflow of the original VAE test, and I was able to reproduce the results, and then not... inconsistent. Then a new fact came up:

TRT engine's first inference after loading incurs a one-time warm-up cost (CUDA kernel selection + scratch memory allocation) that doesn't exist for normal PyTorch models, so any benchmark must discard the first run from both to compare steady-state performance.

Benchmark but discarding first result, reduced test to sampler_scheduler_pairs*2seeds=8:

VAE TRT: Prompt executed in 86.52 seconds
No VAE TRT: Prompt executed in 85.37 seconds

Possible causes:

SDXL VAE is not a significant enough compute to compensate for the TRT init overhead
The XY Plot workflows I am testing in are the worst case scenario for VAE TRT (I reckon TRT shows more benefit on batch>1 compared to batch=1,repeat that XY Plot does)
ComfyUI's implementations have evolved since back then and the speedup is no longer feasible

At least the SDXL LoRA refitting is a success. Applying LoRAs to an already built SDXL engine (inpaint also supported in my fork), now takes 30-35s on an RTX 4060TI 16GB, compared to having to rebuild the engine, which takes 5m+. I have been able to swap LoRAs in my SDXL workflows with comparatively very little notable overhead. For most workflow runs the speed gain of TRT outweighs the refit overhead at the first execution.

Navigation Menu

Repository navigation

Plans of continued development? #121

Update

Chat: "DiT TRT test - ONNX CPU export blocker"

So

ComfyUI TensorRT Fork — Patch Summary

1. ONNX Export Compatibility

2. Step Size Reduction

3. Lazy Engine Load + VRAM Reporting

4. LoRA Refit (UNTESTED)

TensorRT LoRA Refit

Setup (one-time)

Usage (per-LoRA , fast)

LoRA Refit

SDXL VAE TRT

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Plans of continued development? #121

Description

Activity

NubeBuster commented on Jan 31, 2026

jtabox commented on Feb 3, 2026

lord-lethris commented on Feb 18, 2026

NubeBuster commented on Feb 19, 2026

Update

Chat: "DiT TRT test - ONNX CPU export blocker"

So

SveSop commented on Feb 25, 2026

NubeBuster commented on Feb 27, 2026

NubeBuster commented on Mar 30, 2026

ComfyUI TensorRT Fork — Patch Summary

1. ONNX Export Compatibility

2. Step Size Reduction

3. Lazy Engine Load + VRAM Reporting

4. LoRA Refit (UNTESTED)

TensorRT LoRA Refit

Setup (one-time)

Usage (per-LoRA , fast)

NubeBuster commented on Mar 30, 2026

NubeBuster commented on Apr 5, 2026

LoRA Refit

SDXL VAE TRT

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions