About function calling failures using the huggingface transformer
hi.
I've been following the tutorial in Qwen's documentation.
(https://qwen.readthedocs.io/en/latest/framework/function_call.html#hugging-face-transformers)
I noticed that the function calling using the hugging face transformer is not working properly like:
To provide you with the current temperature in San Francisco, I will use the get_current_temperature tool. For the temperature forecast for tomorrow, I will need to use a weather forecasting API, which is not listed among the available tools. I can still fetch today's temperature and assume it as an estimate for tomorrow since no specific tool is provided for future dates. Let's start with today's temperature.
{"name": "get_current_temperature", "arguments": {"location": "San Francisco, California, USA"}}
<|im_end|>
I think parsing JSON is the problem. But I don't know what to do.
Hi, I test the case with the same model provided by the DashScope API, and it is working fine. Could you share your script?
Hi.
The functions 'get_current_temperature' and 'get_temperature_date', as well as 'TOOLS' and 'MESSAGES' are the same as in the Qwen documentation.
def get_current_temperature(location: str, unit: str = "celsius"):
"""Get current temperature at a location.
Args:
location: The location to get the temperature for, in the format "City, State, Country".
unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])
Returns:
the temperature, the location, and the unit in a dict
"""
return {
"temperature": 26.1,
"location": location,
"unit": unit,
}
def get_temperature_date(location: str, date: str, unit: str = "celsius"):
"""Get temperature at a location and date.
Args:
location: The location to get the temperature for, in the format "City, State, Country".
date: The date to get the temperature for, in the format "Year-Month-Day".
unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"])
Returns:
the temperature, the location, the date and the unit in a dict
"""
return {
"temperature": 25.9,
"location": location,
"date": date,
"unit": unit,
}
TOOLS = [
{
"type": "function",
"function": {
"name": "get_current_temperature",
"description": "Get current temperature at a location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": 'The location to get the temperature for, in the format "City, State, Country".',
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": 'The unit to return the temperature in. Defaults to "celsius".',
},
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "get_temperature_date",
"description": "Get temperature at a location and date.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": 'The location to get the temperature for, in the format "City, State, Country".',
},
"date": {
"type": "string",
"description": 'The date to get the temperature for, in the format "Year-Month-Day".',
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": 'The unit to return the temperature in. Defaults to "celsius".',
},
},
"required": ["location", "date"],
},
},
},
]
MESSAGES = [
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\nCurrent Date: 2024-09-30"},
{"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"},
]
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_name_or_path = "./models/Qwen/Qwen2.5-14B-Instruct-1M"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="float16",
)
model = AutoModelForCausalLM.from_pretrained(
model_name_or_path,
device_map="auto",
quantization_config=bnb_config,
)
tools = TOOLS
# tools = [get_current_temperature, get_temperature_date]
messages = MESSAGES[:]
text = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
output_text = tokenizer.batch_decode(outputs)[0][len(text):]
print(output_text)
@NYAGU The issue might be due to the 4-bit quantization. I obtained the correct result when I disabled the quantization.
<tool_call>
{"name": "get_current_temperature", "arguments": {"location": "San Francisco, California, USA"}}
</tool_call>
<tool_call>
{"name": "get_temperature_date", "arguments": {"location": "San Francisco, California, USA", "date": "2024-10-01"}}
</tool_call><|im_end|>