batch size will affect llm inference results? #1250

System Info

GPU：A30*1
system: Ubuntu 22.04
trtllm: 0.7.1
tritonserver: 23.12
model: qwen 7b/14b

Who can help?

@byshiue

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1

python build.py --hf_model_dir ./model_zoon/Qwen-7B-Chat/
--dtype float16
--remove_input_padding
--use_gpt_attention_plugin float16
--enable_context_fmha
--use_inflight_batching
--paged_kv_cache
--use_gemm_plugin float16
--output_dir ./model_zoon/Qwen-7B-Chat-fp16-trt/

2 build model repository

trtllm_bls+trtllm+preprocess+postprocess
inflight batching enable

3. launch server

python3 scripts/launch_triton_server.py --model_repo=tensorrtllm_backend-0.7.1/trt-models-repository-2/ --metrics_port=8005 --world_size=1

4 test

q1 = {
"stream":false,
"max_tokens": 1000,
"bad_words": "",
"stop_words": "<|im_end|>",
"pad_id":151645,
"end_id":151645,
"top_k":1,
"presence_penalty": 1.05,
"text_input": "<|im_start|>\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n北京的夏天是什么样子<|im_end|>\n<|im_start|>assistant\n"
}
q2={
"stream":false,
"max_tokens": 1000,
"bad_words": "",
"stop_words": "<|im_end|>",
"pad_id":151645,
"end_id":151645,
"top_k":1,
"text_input": "<|im_start|>\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n北京的春天是什么样子<|im_end|>\n<|im_start|>assistant\n"
}
First, initiate program A which comprises four threads concurrently sending the following request q1 in a continuous loop for 1000 iterations. Simultaneously, initiate program B to send the following request q2. In theory, the response from program B should be consistent every time, but in practice, program B occasionally produces varying results.

Expected behavior

Theoretically, when topk=1, the response to each request should be the same. However, in reality, there are occasionally a few characters that are different.

"text_output":"北京的春天一般在3月中旬到5月中旬，这段时间的天气温暖宜人，白天和晚上的温差较大，白天最高气温一般在20-25摄氏度，晚上最低气温一般在10-15摄氏度。春天的北京，万物复苏，花儿盛开，树木发芽，鸟儿鸣叫，是一年中最美的季节之一。"

actual behavior

In practice, the following two types of responses will be received.
First:
"text_output":"北京的春天一般在3月中旬到5月中旬，这段时间的天气温暖宜人，白天和晚上的温差较大，白天最高气温一般在20-25摄氏度，晚上最低气温一般在10-15摄氏度。春天的北京，万物复苏，花儿盛开，树木发芽，鸟儿鸣叫，是一年中最美的季节之一。"
second:
"text_output":"北京的春天一般在3月中旬到5月中旬，这段时间的天气温暖宜人，白天和晚上的温差较大，白天最高气温一般在20-25摄氏度，晚上最低气温一般在10-15摄氏度。春天是北京最美的季节之一，此时的北京公园、街道、胡同等地方都充满了生机和活力，各种花卉盛开，景色宜人。"}

additional notes

I suspect it's because floating-point calculations do not obey the commutative law. Requests from different batches may have varying block sizes in GEMM, potentially leading to slight differences in GEMM results and ultimately causing variations in inference outcomes. However, I don't have any evidence to support this, haha.