You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Difference between max_batch_size in the engine builder and max_num_sequences in TrtGptModelOptionalParams? #65
Hi @michaelroyzen , I'm copying the answer I've got from one of the engineers working on the batch manager:
max_num_sequences is the maximum number of requests/sequences the backend can maintain state (KV cache, decoder state) for. It could be smaller, equal or larger than the TensorRT engine max_batch_size. If not specified, it will be set to max_batch_size if enable_trt_overlap= False, and set to 2*max_batch_size if enable_trt_overlap=True (to have at least two batches to ping-pong between). The scheduler tries to form a batch of size min(max_num_sequences, max_batch_size), making sure that max_num_tokens is also satisfied. Currently, the requests are scheduled based on the order they were received. In the future, we might have more scheduling options, allowing the scheduler to choose from any the max_num_sequences requests when creating the batch.
Thanks, Julien. I'm having problems with an 8x A100 instance being totally unresponsive when it gets saturated with requests. I am running max_batch_size 8 and max_num_sequences 64.
So if I understand correctly, if I suddenly get 64 requests the scheduler will make 8 batches of 8 and round-robin their execution? Is that what enable_trt_overlap does? That's what I'd expect it to do.
Furthermore, how do I debug and see metrics for how many batches are inflight at a given time? There is no info about this in the standard Triton metrics. @jdemouth-nvidia
This is a pretty urgent issue -- we deployed TRT-LLM in production today with the Triton backend and are facing these errors in real-time. I'd also appreciate general pointers on how to optimize our config for high throughput on 8x A100/H100.
Update: at some intermediate level of traffic before total failure, the runtime will begin returning a single token over and over again. This has been reported by multiple different users and reproduced myself.
TRT-LLM with the Triton backend is not currently usable for us under load -- first it just starts printing the token "meteor" over and over for every user then it stops responding whatsoever until load decreases. However, we currently have no way of actually assessing real time load. What is causing these issues and how can we debug them more effectively @jdemouth-nvidia?
We know this is not an issue with the model because it generates correct output with vLLM and Huggingface runtimes.
I'd also like to get metrics about the number of in flight requests and KV cache utilization.
Hi @michaelroyzen , we pushed an update to the main branch, including a fix to Multi-block MMHA, which should solve the issue that the token "meteor" is printed over and over again. Please let us know if there still are issues after the fix, thanks.
Activity
jdemouth-nvidia commentedon Oct 23, 2023
Hi @michaelroyzen , I'm copying the answer I've got from one of the engineers working on the batch manager:
max_num_sequencesis the maximum number of requests/sequences the backend can maintain state (KV cache, decoder state) for. It could be smaller, equal or larger than the TensorRT enginemax_batch_size. If not specified, it will be set tomax_batch_sizeifenable_trt_overlap= False, and set to2*max_batch_sizeifenable_trt_overlap=True(to have at least two batches to ping-pong between). The scheduler tries to form a batch of sizemin(max_num_sequences, max_batch_size), making sure thatmax_num_tokensis also satisfied. Currently, the requests are scheduled based on the order they were received. In the future, we might have more scheduling options, allowing the scheduler to choose from any themax_num_sequencesrequests when creating the batch.I hope it answers your question. Thanks, Julien
michaelroyzen commentedon Oct 24, 2023
Thanks, Julien. I'm having problems with an 8x A100 instance being totally unresponsive when it gets saturated with requests. I am running
max_batch_size8 andmax_num_sequences64.So if I understand correctly, if I suddenly get 64 requests the scheduler will make 8 batches of 8 and round-robin their execution? Is that what
enable_trt_overlapdoes? That's what I'd expect it to do.Furthermore, how do I debug and see metrics for how many batches are inflight at a given time? There is no info about this in the standard Triton metrics. @jdemouth-nvidia
This is a pretty urgent issue -- we deployed TRT-LLM in production today with the Triton backend and are facing these errors in real-time. I'd also appreciate general pointers on how to optimize our config for high throughput on 8x A100/H100.
Update: at some intermediate level of traffic before total failure, the runtime will begin returning a single token over and over again. This has been reported by multiple different users and reproduced myself.
michaelroyzen commentedon Oct 24, 2023
TRT-LLM with the Triton backend is not currently usable for us under load -- first it just starts printing the token "meteor" over and over for every user then it stops responding whatsoever until load decreases. However, we currently have no way of actually assessing real time load. What is causing these issues and how can we debug them more effectively @jdemouth-nvidia?
We know this is not an issue with the model because it generates correct output with vLLM and Huggingface runtimes.
I'd also like to get metrics about the number of in flight requests and KV cache utilization.
Thanks again for your help.
jdemouth-nvidia commentedon Oct 24, 2023
Hi @michaelroyzen - let me see what we can do to debug that problem more efficiently. Thanks.
kaiyux commentedon Nov 7, 2023
Hi @michaelroyzen , we pushed an update to the main branch, including a fix to Multi-block MMHA, which should solve the issue that the token "meteor" is printed over and over again. Please let us know if there still are issues after the fix, thanks.
byshiue commentedon Dec 11, 2023
Close this bug because the issue is inactivated. Feel free to ask here if you still have question/issue, we will reopen the issue.