Function Calling and Tool Use in LLMs

Large Language Models (LLMs) are powerful text generators, but their knowledge is frozen at the time of their training and they cannot, by themselves, interact with the outside world. Function calling, also known as tool use, is the critical mechanism that bridges this gap. It allows an LLM to request the execution of external code, enabling it to access real-time information, interact with APIs, and perform actions in other software systems. This transforms the LLM from a static knowledge base into a dynamic agent capable of solving complex, multi-step problems.

Function calling is a capability of certain LLMs^[1] where the model, in response to a prompt, can generate a structured data object that specifies a function to be called and the arguments to use. It's crucial to understand that the LLM does not execute the function itself. Instead, your application code receives this structured request, runs the corresponding function (like calling a weather API or querying a database), and then passes the result back to the LLM. The model then uses this new information to generate a final, more accurate and context-aware response to the user. This process effectively gives the LLM access to "tools" it can use to accomplish tasks beyond simple text generation.

How It Works: The Mechanism

The process is a multi-step conversational loop between your application and the LLM API. It is not a single-shot request. Models that support tool use have been specifically fine-tuned to recognize when a tool is needed and to output syntax that conforms to the developer's specifications.

Let's break down the six steps of a typical function call flow.

Step 1: Tool Declaration

First, the developer declares a set of available tools to the LLM. This happens at the beginning of the interaction, typically as part of the initial API call. These declarations are structured descriptions of your functions, including the function's name, a description of what it does, and a schema for its parameters. This schema is almost always defined using JSON Schema^[2], a standard for describing the structure of JSON^[3] data. A good description is vital; the LLM uses the description's semantics to decide when the tool is appropriate.

Step 2: User Prompt and Inference

The user provides a prompt. Your application sends this prompt to the LLM, along with the set of tool declarations from Step 1. The model then performs inference, analyzing the user's intent. Based on its training, it decides if it can answer from its internal knowledge or if it needs to use one of the provided tools to fulfill the request.

Step 3: Tool Call Generation

If the LLM determines a tool is needed, its API response will not contain a message for the user. Instead, it will contain a special tool-call object. This object is a structured request, typically a JSON object, that specifies the name of the function to call and the arguments it has extracted from the user's prompt. For example, for the prompt "What's the weather in Boston?", the model might generate an object like: {"name": "get_weather", "arguments": {"location": "Boston, MA"}}.

Step 4: Application-Side Execution

Your application code receives this tool-call object. It's your code's responsibility to parse this object, validate the function name and arguments, and then execute the actual, corresponding function in your codebase. This could involve making a call to an external Application Programming Interface (API)^[4], running a database query, or executing any other piece of code. This step is the "firewall": the LLM only suggests the call; your code controls the execution and environment.

Step 5: Submitting the Tool's Output

After your function executes, it will produce a result (e.g., a JSON payload from a weather API) or an error. Your application then makes another API call back to the LLM. This time, you pass the entire conversation history, including the user's initial prompt, the model's tool-call request, and a new message containing the output from your function. This output is sent as the result of the tool call.

Step 6: Final Response Generation

The LLM receives the tool's output. It now has the specific information it needed (e.g., "72 degrees and sunny"). It combines this new information with the original user prompt and generates a final, natural-language response (e.g., "The weather in Boston is currently 72 degrees and sunny."). This response is sent back to your application to be displayed to the user.

Some modern LLMs also support parallel function calling. This allows the model to request multiple tool calls in a single turn, which your application can then execute concurrently. This is particularly useful for complex queries that require information from multiple independent sources, as it can significantly reduce overall latency.

A Brief History and Context

The idea of connecting language models to external tools is not new, with roots in early research on intelligent agents^[5] and symbolic reasoning systems. However, the practical, large-scale implementation of this capability in commercial LLMs is a more recent development.

A key conceptual framework is the "Reason, Act" (ReAct) paradigm, proposed by researchers at Google Brain^[6] in a 2022 paper^[7]. This approach demonstrated that LLMs could achieve greater performance on complex tasks by generating both reasoning traces (thought) and actions for an external environment.

The commercial popularization of this feature began in earnest in 2023. OpenAI announced function calling^[8] for its GPT models in June 2023, providing a formal API for developers to describe functions to the model. This was a significant step, as it moved the capability from a prompt-engineering hack to a first-class feature of the model, backed by specific fine-tuning. OpenAI later enhanced this with "parallel function calling" at its first developer conference in November 2023^[9], allowing for more efficient data retrieval.

Following this, other major providers rolled out their own versions. Google integrated tool use into its Vertex AI platform for its Gemini family of models. Anthropic also introduced tool use for its Claude models^[10], adopting a similar pattern of function declaration and response. The de facto standard for defining the structure of these functions has become JSON Schema, reflecting its flexibility and widespread adoption in the API ecosystem.

A Concrete Example: Weather and Calendar

Let's walk through a practical example using Python and a hypothetical LLM client. We want to build an assistant that can answer: "What is the weather like in New York City, and what is on my calendar for tomorrow?"

This requires two distinct tools: one for weather and one for calendar events.

1. Defining the Tools in Code

First, we'd have our Python functions that do the real work. These are just standard functions; they don't know anything about the LLM.

import json
import datetime

# Dummy functions for demonstration
def get_current_weather(location: str, unit: str = "fahrenheit"):
 """Get the current weather in a given location."""
 if "new york" in location.lower():
 return json.dumps({
 "location": "New York City",
 "temperature": "58",
 "unit": unit,
 "forecast": "partly cloudy"
 })
 return json.dumps({"location": location, "temperature": "unknown"})

def get_calendar_events(start_date: str, end_date: str):
 """Get calendar events for a given date range."""
 return json.dumps([
 {"event": "Team Standup", "time": "09:00 AM"},
 {"event": "Project Phoenix Sync", "time": "11:30 AM"},
 {"event": "Dentist Appointment", "time": "03:00 PM"}
 ])

2. Describing the Tools for the LLM

Next, we describe these functions to the LLM using a JSON Schema-like structure. This is the critical declaration step.

tools_description = [
 {
 "type": "function",
 "function": {
 "name": "get_current_weather",
 "description": "Get the current weather in a given location.",
 "parameters": {
 "type": "object",
 "properties": {
 "location": {
 "type": "string",
 "description": "The city and state, e.g., San Francisco, CA",
 },
 "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
 },
 "required": ["location"],
 },
 },
 },
 {
 "type": "function",
 "function": {
 "name": "get_calendar_events",
 "description": "Get all calendar events for a given date range.",
 "parameters": {
 "type": "object",
 "properties": {
 "start_date": {
 "type": "string",
 "description": "The start date in YYYY-MM-DD format.",
 },
 "end_date": {
 "type": "string",
 "description": "The end date in YYYY-MM-DD format.",
 },
 },
 "required": ["start_date", "end_date"],
 },
 },
 }
]

3. The Multi-Turn Conversation

Here's how the interaction with the LLM API would unfold, managed by our application code.

Turn 1: User Prompt

• App sends to LLM:

• User prompt: "What is the weather like in New York City, and what is on my calendar for tomorrow?"

• Tool descriptions: The tools_description list from above.

• LLM responds with (conceptual): A request to call both functions in parallel.

{
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\": \"New York City\"}"
}
},
{
"id": "call_def456",
"type": "function",
"function": {
"name": "get_calendar_events",
"arguments": "{\"start_date\": \"2024-10-27\", \"end_date\": \"2024-10-27\"}"
}
}
]
}

The article is dated. This example assumes a 'today' of October 26, 2024, which is nearly two years in the past from the current date of October 2026. This makes the example confusing and appear out of date.

Turn 2: Execution and Response

• App logic:

1. Parses the tool_calls array.
2. Sees two calls and executes them. Because they are independent, it can run them concurrently.
3. get_current_weather("New York City") returns {"location": "New York City", ...}.
4. get_calendar_events(...) returns [{"event": "Team Standup", ...}].
5. Collects the results.

• App sends to LLM: The full history, plus the results from the tool calls.

[
{ "role": "user", "content": "What is the weather...?" },
{ "role": "assistant", "tool_calls": [...] }, // from previous turn
{
"role": "tool",
"tool_call_id": "call_abc123",
"content": "{\"location\": \"New York City\", \"temperature\": \"58\", \"unit\": \"fahrenheit\", \"forecast\": \"partly cloudy\"}"
},
{
"role": "tool",
"tool_call_id": "call_def456",
"content": "[{\"event\": \"Team Standup\", \"time\": \"09:00 AM\"}, ...]"
}
]

Turn 3: Final Answer

• LLM responds with: A final, synthesized natural language answer for the user.

"The current weather in New York City is 58 degrees Fahrenheit and partly cloudy. For tomorrow, you have three events on your calendar: Team Standup at 9:00 AM, Project Phoenix Sync at 11:30 AM, and a Dentist Appointment at 3:00 PM."

This example illustrates the power of tool use: the LLM acts as a reasoning engine to decompose a complex query, dispatch requests for information, and synthesize the results into a single, coherent answer.

The Decision Lens: Tradeoffs, Costs, and Failure Modes

While powerful, function calling is not a silver bullet. Technical leaders must weigh its benefits against its inherent complexities and costs.

When to Use Tool Use

• Accessing Real-Time Data: Use it when answers depend on information created after the model's training cut-off, such as stock prices, news headlines, or weather forecasts.
• Interacting with Private Data: It's the primary mechanism for allowing an LLM to answer questions about a company's internal knowledge base, user data, or databases. The LLM calls a function that you control, which then securely accesses the private data.
• Performing Actions: When you need the LLM to trigger an action in the world, like sending an email, booking a meeting, creating a ticket in Jira^[11], or updating a Salesforce^[12] record.
• Structured Output: Even without calling a function, you can use the tool-calling mechanism to force the LLM to return structured JSON data that conforms to a predefined schema. This is a reliable way to get predictable, machine-readable output.

When to Reconsider

• Low-Latency Requirements: Each tool use cycle involves at least one additional round trip to the LLM API, adding significant latency (hundreds of milliseconds to several seconds). For applications requiring near-instantaneous responses, this may be unacceptable.
• Simple, Self-Contained Tasks: If a task is purely creative writing or answering a general knowledge question that doesn't require live data, tool use adds unnecessary complexity and cost.
• Cost Sensitivity: Function calling increases costs. You pay for the initial call, the call with the tool output, and the tokens used for the function definitions and their results. A single user query can easily turn into 3 or 4 separate, token-heavy API calls. Cloud provider pricing pages^[13] for their AI models show how input and output tokens are billed differently, and tool use generates a lot of both.

Key Tradeoffs and Failure Modes

A system built on LLM tool use is a probabilistic one. You must design your application defensively.

• Incorrect Tool Selection: The model might choose the wrong tool for a given query, or fail to use a tool when it should. Clear, distinct, and well-written function descriptions are your primary defense here.
• Argument Hallucination: The model might invent arguments, provide them in the wrong format, or fail to extract them correctly from the prompt. Your code must rigorously validate all arguments received from the LLM before executing any function. Never trust the input.
• Execution Errors: The tool itself might fail. An external API could be down, a database query could time out, or a user might lack permissions. Your code must catch these errors and return a clear error message as the tool's output. The LLM can often interpret this error and either try again or inform the user gracefully.
• Security: This is paramount. Since the LLM is generating requests to run your code, you must treat these requests as untrusted user input. A user could craft a prompt attempting to trick the LLM into calling a destructive function (e.g., "delete all my user data"). Your application's execution layer is the final security gatekeeper. It should enforce permissions and never expose functions that could be abused. For example, never give an LLM a tool that can directly execute arbitrary shell commands.

Correcting Common Misconceptions

"The LLM is executing my code."

This is the most common and dangerous misconception. The LLM never has direct access to your environment or your code. It only generates a JSON object suggesting that your code should be run. Your application is the one that interprets this suggestion and decides whether and how to act on it. This separation is a critical security boundary.

"It's just a fancy prompt."

While prompt engineering is part of the equation (especially in the function descriptions), the tool-use feature runs deeper. Models with this capability have undergone specific fine-tuning^[14] to better understand the intent to use a tool and to reliably generate the required structured output. This is why a model with native function calling is far more reliable at the task than trying to coax a standard model to generate JSON through prompting alone.

"Function calling is always deterministic and reliable."

Absolutely not. The LLM's decision to call a function, which function to call, and what arguments to use are all probabilistic predictions. It can and will make mistakes. Robust application design involves anticipating these failures through validation, error handling, and potentially including retry logic or asking the user for clarification.

The State of Tool Use Today

This entire section is outdated. The article is framed from a "late 2024" perspective, but it is currently late 2026. The field has advanced significantly, with new models like GPT-5 and GPT-6, and open-weight models from companies like Mistral and Meta having much more mature tool-use capabilities than what is described.

• Providers: OpenAI's GPT-4, Google's Gemini models^[15], and Anthropic's Claude 3 family^[16] all offer robust, first-party support for tool use. Their APIs have converged on a similar pattern: declare tools with JSON Schema, receive a tool call, execute, and return the result.
• Standardization: While no formal IETF^[17] or W3C^[18] standard exists for LLM tool use, JSON Schema^[19] has emerged as the de facto standard for describing the function signatures. This convergence simplifies development, as the same schema can often be used across different model providers with minor adjustments.
• Open Source: The open-source community has also embraced tool use. Models like those from Mistral AI^[20] and Meta's Llama series are increasingly being fine-tuned by the community and companies to perform tool-calling tasks. Frameworks like LangChain^[21] and LlamaIndex^[22] provide higher-level abstractions over these APIs, simplifying the development of complex, multi-tool agents.
• Frontiers: The cutting edge of this technology involves improving the LLM's reasoning capabilities. This includes more complex multi-step tool-use chains, better error correction (e.g., the model automatically retrying a failed API call with different parameters), and giving the model the ability to dynamically choose from hundreds or thousands of potential tools. As models become more capable, the line between a simple tool-using application and a fully autonomous agent will continue to blur.

This capability is fundamental to building sophisticated AI-powered applications. By understanding its mechanics, tradeoffs, and failure modes, technical leaders can make informed decisions about how and when to connect their systems to the powerful reasoning engine of a large language model.