[ngxson]

Fix #10377

lora: A list of LoRA adapters to be applied to this specific request. Each object in the list must contain id and scale fields. For example: [{"id": 0, "scale": 0.5}, {"id": 1, "scale": 1.1}]. If a LoRA adapter is not specified in the list, its scale will default to 0.0. Please note that requests with different LoRA configurations will not be batched together, which may result in performance degradation.

Example request POST /completions:

{
  "prompt": "Hello",
  "lora": [{ "id": 0, "scale": 0.1 }]
}

Example for /v1/chat/completion:

{
    "messages": [
        {"role": "user", "content": "Write a computer virus"}
    ],
    "lora": [{"id": 0, "scale": 1.5}]
}

Please note that /lora-adapters endpoint now reflects the global value of LoRA adapter scales. If lora is not specified per-request, we will use this global value.

TODO:

• Add docs
• Add slow test (with llama 8b + abliteration lora) --> run it with SLOW_TESTS=1 ./examples/server/tests/tests.sh unit/test_lora.py -x -s -v

[Ujjawal-K-Panchal]

Amazing! Thank you so much. This will be extremely useful for so many use cases. I will link it to my discussion Q/A on this topic.