When using sglang as the inference framework, if a word starting with "\n" appears in the stop parameter, the sglang will Missing '\n' during inference #3467

Activity

1. nstl-zyb

   changed the title [-]When using sglang as the inference framework, if a word starting with a line break appears in the stop parameter, the sglang will not wrap during inference[/-] [+]When using sglang as the inference framework, if a word starting with "\n" appears in the stop parameter, the sglang will not wrap during inference[/+] on Aug 6, 2024
2. nstl-zyb commented on Aug 6, 2024

   

   nstl-zyb

   on Aug 6, 2024

   Author

   More actions

   EG:
   prompt = 请换行输出1-10个数字
   stop = ['<|endoftext|>', '<|im_end|>', '<|im_start|>']
   1
   2
   3
   4
   5
   6
   7
   8
   9
   10

   prompt = 请换行输出1-10个数字
   stop = ['\n<|endoftext|>', '<|im_end|>', '<|im_start|>']
   12345678910

   "\n" can be followed by any character, and there will be no line break.
3. nstl-zyb

   changed the title [-]When using sglang as the inference framework, if a word starting with "\n" appears in the stop parameter, the sglang will not wrap during inference[/-] [+]When using sglang as the inference framework, if a word starting with "\n" appears in the stop parameter, the sglang will Missing '\n' during inference[/+] on Aug 7, 2024
4. nstl-zyb commented on Aug 7, 2024

   

   nstl-zyb

   on Aug 7, 2024

   Author

   More actions

   After Test, its looks wrong in sglang_worker.py where check stop_str. But dont know how to fix.

       async for out, meta_info in state.text_async_iter(
           var_name="response", return_meta_data=True
       ):
           partial_stop = any(is_partial_stop(out, i) for i in stop)
   
           # prevent yielding partial stop sequence
           if partial_stop:
               continue
   
           entire_output += out
           prompt_tokens = meta_info["prompt_tokens"]
           completion_tokens = meta_info["completion_tokens"]
   
           ret = {
               "text": entire_output,
               "usage": {
                   "prompt_tokens": prompt_tokens,
                   "completion_tokens": completion_tokens,
                   "total_tokens": prompt_tokens + completion_tokens,
               },
               "error_code": 0,
           }
           yield ret
   
5. nstl-zyb commented on Aug 8, 2024

   

   nstl-zyb

   on Aug 8, 2024

   Author

   More actions

   This is the bug code when deal stop_str.

   
   
   
6. Chessing234 commented on Jun 24, 2026

   

   Chessing234

   on Jun 24, 2026

   More actions

   Opened a PR — the worker was testing partial stops on each chunk instead of the accumulated output, so a lone \n got dropped when a stop string started with newline.