toolcalling with vllm is broken

[suresh-now]

Question

I am following the example in https://ai.pydantic.dev/examples/rag/#example-code

I am also using the latest version of vllm (0.8.3) with toolcalling support with --guided_decoding_backend.

The rag example doesnt work with any of models listed in https://docs.vllm.ai/en/stable/features/tool_calling.html#llama-models-llama3-json. Some models (llama3.2) correctly returns the tool call, but agent is unable to parse and make a call tool

I used llama3.1 https://docs.vllm.ai/en/latest/features/tool_calling.html#llama-models-llama3-json, the tool call happens repeatedly (never ending), so the token limit exceed, and crashes.

However, using openai model works perfectly. Even ollama works (without streaming)

Additional Context

Pydantic AI version - pydantic-ai==0.0.43
Python - 3.11.0
vllm - 0.0.83 - vllm-project/vllm#13483 (this doesnt fix the issue I am facing)

[aristideubertas]

In my experience tool calling with models such as Qwen and LLAMA hasn't worked very where since it seems like they do not natively support function calling, and some sort of parsing has to happen for the tool usage to happen. I think VLLM is supposed to fix this internally with certain options, but I wonder if anything can be done on Pydantic side to support these models.

[ItzAmirreza]

I'm experiencing the same issue. It calls the tools again and again until it runs out of context and timesout. It wasn't a problem a version ago but after I updated my pydantic ai, it doesn't work anymore. Something has definitely changed, but still investigating. Please update me if you found any fix.

In my experience tool calling with models such as Qwen and LLAMA hasn't worked very where since it seems like they do not natively support function calling, and some sort of parsing has to happen for the tool usage to happen. I think VLLM is supposed to fix this internally with certain options, but I wonder if anything can be done on Pydantic side to support these models.

As far as I know Qwen2.5 is specifically trained for agentic use, even in their model card it states about tool calling and function calling.

[webcoderz]

Related to: #825