[bartowski1182]

After referencing:

https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407
https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503

and how Qwen does tool calling, plus some help from my good friend Claude, was able to figure out proper tool calling support and what seems to be a proper chat template that follows the best practices from mistral themselves

Hosted with this command:

./llama-server -m /models/Mistral-Small-3.2-24B-Instruct-2506-GGUF/mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf -ngl 100 --host 0.0.0.0 -fa --slots --jinja --chat-template-file /models/Mistral-Small-3.2-24B-Instruct-2506.jinja

Used the following script to validate:

test_mistral.py script for checking the tool use works

#!/usr/bin/env python3
"""
Tool calling test script for llama.cpp OpenAI-compatible endpoint
Tests the full flow: initial request -> tool call -> mock result -> final response
"""

import json
import requests
from typing import Dict, Any, List

# Configuration
API_URL = "http://localhost:8080/v1/chat/completions"
MODEL = "mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf"

# Tool definition
WEATHER_TOOL = {
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and country/state, e.g. `San Francisco, CA`, or `Paris, France`"
                }
            },
            "required": ["location"]
        }
    }
}

def make_request(messages: List[Dict[str, Any]], tools: List[Dict[str, Any]] = None) -> Dict[str, Any]:
    """Make a request to the API"""
    payload = {
        "model": MODEL,
        "messages": messages
    }
    
    if tools:
        payload["tools"] = tools
    
    response = requests.post(API_URL, json=payload)
    response.raise_for_status()
    return response.json()

def get_mock_weather(location: str) -> Dict[str, Any]:
    """Generate mock weather data for a location"""
    # Mock weather data
    mock_data = {
        "Istanbul": {
            "temperature": 18,
            "condition": "partly cloudy",
            "humidity": 65,
            "wind_speed": 12
        },
        "default": {
            "temperature": 22,
            "condition": "sunny",
            "humidity": 50,
            "wind_speed": 8
        }
    }
    
    weather = mock_data.get(location.split(",")[0].strip(), mock_data["default"])
    
    return {
        "location": location,
        "temperature_celsius": weather["temperature"],
        "condition": weather["condition"],
        "humidity_percent": weather["humidity"],
        "wind_speed_kmh": weather["wind_speed"]
    }

def main():
    print("🚀 Testing tool calling with llama.cpp endpoint\n")
    
    # Initial conversation
    messages = [
        {"role": "system", "content": "You are a chatbot that uses tools/functions. Don't overthink things."},
        {"role": "user", "content": "What is the weather in Istanbul?"}
    ]
    
    print("📤 Sending initial request with tool definition...")
    print(f"User: {messages[-1]['content']}\n")
    
    # Step 1: Initial request with tool
    try:
        response = make_request(messages, [WEATHER_TOOL])
        print("📥 Received response:")
        print(json.dumps(response, indent=2))
        print()
        
        # Extract the assistant's response
        assistant_message = response["choices"][0]["message"]
        messages.append(assistant_message)
        
        # Check if the model made a tool call
        if "tool_calls" in assistant_message and assistant_message["tool_calls"]:
            tool_call = assistant_message["tool_calls"][0]
            function_name = tool_call["function"]["name"]
            function_args = json.loads(tool_call["function"]["arguments"])
            
            print(f"🔧 Model called tool: {function_name}")
            print(f"   Arguments: {function_args}\n")
            
            # Step 2: Execute the tool (mock)
            if function_name == "get_current_weather":
                location = function_args.get("location", "Unknown")
                weather_data = get_mock_weather(location)
                
                print(f"🌤️  Mock weather data for {location}:")
                print(json.dumps(weather_data, indent=2))
                print()
                
                # Step 3: Send tool result back to model
                tool_result_message = {
                    "role": "tool",
                    "tool_call_id": tool_call["id"],
                    "content": json.dumps(weather_data)
                }
                messages.append(tool_result_message)
                
                print("📤 Sending tool result back to model...")
                final_response = make_request(messages)
                
                print("📥 Final response from model:")
                print(json.dumps(final_response, indent=2))
                print()
                
                # Extract and display the final answer
                final_content = final_response["choices"][0]["message"]["content"]
                print("💬 Assistant's final answer:")
                print(final_content)
                
                # Add the assistant's response to messages
                messages.append({
                    "role": "assistant",
                    "content": final_content
                })
                
                # Continue with more turns
                print("\n" + "="*60 + "\n")
                test_additional_turns(messages)
            else:
                print(f"❓ Unknown tool called: {function_name}")
        else:
            print("⚠️  Model did not make a tool call")
            if "content" in assistant_message:
                print(f"Model response: {assistant_message['content']}")
                
    except requests.exceptions.RequestException as e:
        print(f"❌ Request error: {e}")
    except json.JSONDecodeError as e:
        print(f"❌ JSON parsing error: {e}")
    except KeyError as e:
        print(f"❌ Unexpected response format, missing key: {e}")
    except Exception as e:
        print(f"❌ Unexpected error: {e}")

def test_additional_turns(messages: List[Dict[str, Any]]):
    """Test additional conversation turns after tool usage"""
    print("🔄 Testing additional conversation turns...\n")
    
    # Turn 1: Follow-up question about the weather (no tool needed)
    print("📍 Turn 1: Follow-up question")
    messages.append({
        "role": "user",
        "content": "Is that good weather for sightseeing?"
    })
    print(f"User: {messages[-1]['content']}\n")
    
    try:
        response = make_request(messages)
        assistant_response = response["choices"][0]["message"]["content"]
        print(f"💬 Assistant: {assistant_response}\n")
        messages.append({
            "role": "assistant",
            "content": assistant_response
        })
    except Exception as e:
        print(f"❌ Error in turn 1: {e}\n")
        return
    
    print("-" * 40 + "\n")
    
    # Turn 2: Different topic (no tool needed)
    print("📍 Turn 2: Different topic")
    messages.append({
        "role": "user",
        "content": "By the way, what's the capital of Turkey?"
    })
    print(f"User: {messages[-1]['content']}\n")
    
    try:
        response = make_request(messages)
        assistant_response = response["choices"][0]["message"]["content"]
        print(f"💬 Assistant: {assistant_response}\n")
        messages.append({
            "role": "assistant",
            "content": assistant_response
        })
    except Exception as e:
        print(f"❌ Error in turn 2: {e}\n")
        return
    
    print("-" * 40 + "\n")
    
    # Turn 3: Another weather request (should trigger tool use again)
    print("📍 Turn 3: Another weather request with tool")
    messages.append({
        "role": "user",
        "content": "What about the weather in Paris, France?"
    })
    print(f"User: {messages[-1]['content']}\n")
    
    try:
        response = make_request(messages, [WEATHER_TOOL])
        assistant_message = response["choices"][0]["message"]
        
        if "tool_calls" in assistant_message and assistant_message["tool_calls"]:
            tool_call = assistant_message["tool_calls"][0]
            function_args = json.loads(tool_call["function"]["arguments"])
            
            print(f"🔧 Model called tool again: {tool_call['function']['name']}")
            print(f"   Arguments: {function_args}\n")
            
            # Get mock weather for Paris
            location = function_args.get("location", "Unknown")
            weather_data = get_mock_weather(location)
            
            print(f"🌤️  Mock weather data for {location}:")
            print(json.dumps(weather_data, indent=2))
            print()
            
            # Send tool result
            messages.append(assistant_message)
            messages.append({
                "role": "tool",
                "tool_call_id": tool_call["id"],
                "content": json.dumps(weather_data)
            })
            
            final_response = make_request(messages)
            final_content = final_response["choices"][0]["message"]["content"]
            print(f"💬 Assistant: {final_content}\n")
            messages.append({
                "role": "assistant",
                "content": final_content
            })
        else:
            print("⚠️  Model did not make a tool call for Paris weather")
            if "content" in assistant_message:
                print(f"Model response: {assistant_message['content']}")
    except Exception as e:
        print(f"❌ Error in turn 3: {e}\n")
        return
    
    print("-" * 40 + "\n")
    
    # Turn 4: Comparison question (no tool needed)
    print("📍 Turn 4: Comparison question")
    messages.append({
        "role": "user",
        "content": "Which city has better weather for visiting, Istanbul or Paris?"
    })
    print(f"User: {messages[-1]['content']}\n")
    
    try:
        response = make_request(messages)
        assistant_response = response["choices"][0]["message"]["content"]
        print(f"💬 Assistant: {assistant_response}\n")
    except Exception as e:
        print(f"❌ Error in turn 4: {e}\n")
    
    print("✅ Multi-turn conversation test complete!")

if __name__ == "__main__":
    main()

Which gave this proper output:

test_mistral.py script results

 python3 test_mistral.py 
🚀 Testing tool calling with llama.cpp endpoint

📤 Sending initial request with tool definition...
User: What is the weather in Istanbul?

📥 Received response:
{
  "choices": [
    {
      "finish_reason": "tool_calls",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "type": "function",
            "function": {
              "name": "get_current_weather",
              "arguments": "{\"location\":\"Istanbul, Turkey.\"}"
            },
            "id": "a0b2a9fb4"
          }
        ]
      }
    }
  ],
  "created": 1750711340,
  "model": "mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf",
  "system_fingerprint": "b5697-ef035803",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 39,
    "prompt_tokens": 120,
    "total_tokens": 159
  },
  "id": "chatcmpl-rdyqGaribJmnaex1JO4tMR46XfvLGQlL",
  "timings": {
    "prompt_n": 100,
    "prompt_ms": 100.287,
    "prompt_per_token_ms": 1.0028700000000002,
    "prompt_per_second": 997.1382133277492,
    "predicted_n": 39,
    "predicted_ms": 904.706,
    "predicted_per_token_ms": 23.197589743589745,
    "predicted_per_second": 43.10792677400172
  }
}

🔧 Model called tool: get_current_weather
   Arguments: {'location': 'Istanbul, Turkey.'}

🌤️  Mock weather data for Istanbul, Turkey.:
{
  "location": "Istanbul, Turkey.",
  "temperature_celsius": 18,
  "condition": "partly cloudy",
  "humidity_percent": 65,
  "wind_speed_kmh": 12
}

📤 Sending tool result back to model...
📥 Final response from model:
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The current weather in Istanbul, Turkey is partly cloudy with a temperature of 18\u00b0C, humidity of 65%, and wind speed of 12 km/h."
      }
    }
  ],
  "created": 1750711341,
  "model": "mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf",
  "system_fingerprint": "b5697-ef035803",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 37,
    "prompt_tokens": 120,
    "total_tokens": 157
  },
  "id": "chatcmpl-3sAqhVS4CS8P3TOwm7ewGK5KVP315kq1",
  "timings": {
    "prompt_n": 100,
    "prompt_ms": 79.474,
    "prompt_per_token_ms": 0.79474,
    "prompt_per_second": 1258.2731459345193,
    "predicted_n": 37,
    "predicted_ms": 790.148,
    "predicted_per_token_ms": 21.355351351351352,
    "predicted_per_second": 46.82667044654925
  }
}

💬 Assistant's final answer:
The current weather in Istanbul, Turkey is partly cloudy with a temperature of 18°C, humidity of 65%, and wind speed of 12 km/h.

============================================================

🔄 Testing additional conversation turns...

📍 Turn 1: Follow-up question
User: Is that good weather for sightseeing?

💬 Assistant: Based on the current weather conditions in Istanbul, it seems to be quite pleasant for sightseeing. A temperature of 18°C (64°F) is comfortable for outdoor activities, and the partly cloudy skies suggest that you won't have to deal with intense sun or heavy rain. Just make sure to dress in layers as the weather can change throughout the day, and perhaps carry a light jacket in case it gets chilly. Enjoy your exploration of Istanbul!

----------------------------------------

📍 Turn 2: Different topic
User: By the way, what's the capital of Turkey?

💬 Assistant: The capital of Turkey is Ankara.

----------------------------------------

📍 Turn 3: Another weather request with tool
User: What about the weather in Paris, France?

🔧 Model called tool again: get_current_weather
   Arguments: {'location': 'Paris, France.'}

🌤️  Mock weather data for Paris, France.:
{
  "location": "Paris, France.",
  "temperature_celsius": 22,
  "condition": "sunny",
  "humidity_percent": 50,
  "wind_speed_kmh": 8
}

💬 Assistant: The current weather in Paris, France is sunny with a temperature of 22°C, humidity of 50%, and wind speed of 8 km/h.

----------------------------------------

📍 Turn 4: Comparison question
User: Which city has better weather for visiting, Istanbul or Paris?

💬 Assistant: Both cities have pleasant weather, but the better option depends on your preferences:

* If you enjoy slightly cooler temperatures and lower humidity, Istanbul might be a better choice with its current temperature of 18°C (64°F) and partly cloudy skies.
* If you prefer warmer, sunnier weather with lower humidity, Paris might be more appealing with its current temperature of 22°C (72°F) and sunny skies.

Since the weather in both cities is generally good for sightseeing, you may want to consider other factors, such as the specific attractions you're interested in, when deciding which city to visit.

✅ Multi-turn conversation test complete!

Only thing left is that the yesterday functionality doesn't work, but I think it's best to try to get a change in to minja to support that

[serhii-nakon]

@bartowski1182 Hello I thank you for this template
But still some issue when I try work with langchain using ChatOpenAI with llama.cpp and this models.
With QWQ 32b and Qwen3 32B works just fine with only jinja parameter but with Mistral no

error

llama-cpp-1   | got exception: {"code":500,"message":"Tool call IDs should be alphanumeric strings with length 9! at row 91, column 40:\n                {%- if not tool_call.id is defined or tool_call.id is not string or tool_call.id|length != 9 %}\n                    {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n                                       ^\n                {%- endif %}\n at row 91, column 21:\n                {%- if not tool_call.id is defined or tool_call.id is not string or tool_call.id|length != 9 %}\n                    {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n                    ^\n                {%- endif %}\n at row 90, column 112:\n                {{- '[TOOL_CALLS]' + tool_call.function.name }}\n                {%- if not tool_call.id is defined or tool_call.id is not string or tool_call.id|length != 9 %}\n                                                                                                               ^\n                    {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n at row 90, column 17:\n                {{- '[TOOL_CALLS]' + tool_call.function.name }}\n                {%- if not tool_call.id is defined or tool_call.id is not string or tool_call.id|length != 9 %}\n                ^\n                    {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n at row 88, column 55:\n        {%- if message.get('tool_calls') %}\n            {%- for tool_call in message.tool_calls %}\n                                                      ^\n                {{- '[TOOL_CALLS]' + tool_call.function.name }}\n at row 88, column 13:\n        {%- if message.get('tool_calls') %}\n            {%- for tool_call in message.tool_calls %}\n            ^\n                {{- '[TOOL_CALLS]' + tool_call.function.name }}\n at row 87, column 44:\n    {%- elif message['role'] == 'assistant' %}\n        {%- if message.get('tool_calls') %}\n                                           ^\n            {%- for tool_call in message.tool_calls %}\n at row 87, column 9:\n    {%- elif message['role'] == 'assistant' %}\n        {%- if message.get('tool_calls') %}\n        ^\n            {%- for tool_call in message.tool_calls %}\n at row 86, column 47:\n        {{- '[/INST]' }}\n    {%- elif message['role'] == 'assistant' %}\n                                              ^\n        {%- if message.get('tool_calls') %}\n at row 61, column 5:\n{%- for message in loop_messages %}\n    {%- if message['role'] == 'system' %}\n    ^\n        {%- if message['content'] is string %}\n at row 60, column 36:\n\n{%- for message in loop_messages %}\n                                   ^\n    {%- if message['role'] == 'system' %}\n at row 60, column 1:\n\n{%- for message in loop_messages %}\n^\n    {%- if message['role'] == 'system' %}\n at row 1, column 1:\n{%- set today = strftime_now(\"%Y-%m-%d\") %}\n^\n{%- set default_system_message = \"You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\\nYour knowledge base was last updated on 2023-10-01. The current date is \" + today + \".\\n\\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.\n","type":"server_error"}

[bartowski1182]

Yes someone else pointed out a similar issue, Mistral seems to aggressively enforce that the call ID be exactly length 9, so if a tool is not doing that it'll error

I don't think it makes sense to alter the template, since Mistral specifically mentions their improvements to tool calling formatting...

You can see where they force it here:

https://github.com/mistralai/mistral-common/blob/main/src%2Fmistral_common%2Fprotocol%2Finstruct%2Fvalidator.py#L365-L366