[Question] Can I build the tritonserver, tensorrtllm_backend and tensorrtllm and then use these build files across servers?

I want to deploy triton + tensorrtllm, due to some constraints I cannot use docker container. I have figured out that I need to build the following repos:

First, am I missing something, or should this give me everything i need to deploy a triton inference server with tensorrt_llm backend? Second, these take a few hours to build so can I build these for a specific linux and CUDA version and use them across servers (all with potentially different GPUs), given that all the servers have the same linux and CUDA version?

Finally when building LLM models using trt_build can I reuse these models across GPUs or if built on H100, they can only be used on H100?

Thank you for all your work!

Yes, you should be able to build for a specific linux and CUDA version and use across servers with same linux and CUDA version.

However, the TRT-LLM engines generated by trtllm-build are optimized for a specific GPU type and cannot be reused across different GPUs.

@pcastonguay

Thank you @pcastonguay for getting back to me. Just to confirm, if I build all of these:
https://github.com/triton-inference-server/server
https://github.com/NVIDIA/TensorRT
https://github.com/NVIDIA/TensorRT-LLM
https://github.com/triton-inference-server/tensorrtllm_backend

I should have everything to deploy triton + tensorrt_llm_backend, right? Am I missing some steps?

This should give you everything you need.

Great! Thank you.

@pcastonguay

Hey @pcastonguay I have a follow up question:

Can I take the built files from inside the triton container, and use those? Instead of building it myself? What would that look like, is it even possible?

From inside the container, I can see:

root@__:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

So I can use the build files from the container directly, as long as the CUDA version matches, and its Ubuntu 22.04.4, right?

Yes that should work if needed shared libraries versions match too.

@pcastonguay

I tried just using tar to zip the / and recreating it completely (as a best cause scenario), it bugged out because the new server's driver version didn't match exactly. So I am guessing the driver version should exactly match too? @pcastonguay

@chrisreese-if

@chrisreese-if , Just checking in to see if this issue is still relevant for you. If not, please let me know, so I can close it.
By the way, I think your guess was generally right~ :)

Issue has not received an update in over 14 days. Adding stale label.

This issue was closed because it has been 14 days without activity since it has been marked as stale.

Navigation Menu

Repository navigation

[Question] Can I build the tritonserver, tensorrtllm_backend and tensorrtllm and then use these build files across servers? #2423

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Question] Can I build the tritonserver, tensorrtllm_backend and tensorrtllm and then use these build files across servers? #2423

Description

Activity

pcastonguay commented on Nov 12, 2024

chrisreese-if commented on Nov 12, 2024

pcastonguay commented on Nov 12, 2024

chrisreese-if commented on Nov 12, 2024

chrisreese-if commented on Nov 15, 2024

chrisreese-if commented on Nov 15, 2024

pcastonguay commented on Nov 18, 2024

chrisreese-if commented on Nov 19, 2024

karljang commented on Dec 1, 2025

github-actions commented on Dec 16, 2025

github-actions commented on Dec 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions