How to Build an AI Agent: The Loop, Memory, and Tools

Matt George

Fractional CTO · Founder of Injuria.ai

•13 min read•

An AI agent is not a new type of model. It's a system architecture that uses a Large Language Model (LLM)[1] as a reasoning engine to achieve a goal. Unlike a simple chatbot that responds to a single prompt, an agent can perform a sequence of actions, observe the results, and adapt its strategy over multiple steps to complete complex tasks. This capability is enabled by a simple but powerful construct: a loop that combines the LLM's reasoning with external tools and a memory system.

This architecture allows an agent to move beyond just generating text and interact with the world. It can browse the web, query databases, execute code, and call any available API[2]. Understanding how this system works, including its components and significant limitations, is critical for any technical leader deciding where and how to apply this technology. This is not about sentience; it's about a programmatic loop that orchestrates LLM calls to simulate a reasoning process.

What is an AI Agent? The Core Loop Explained

The core of an AI agent is a control loop that enables it to plan and execute tasks autonomously. This concept was heavily influenced by research papers like "ReAct: Synergizing Reasoning and Acting in Language Models"[3], which demonstrated that LLMs could generate both reasoning traces and task-specific actions. The agent loop breaks down a complex goal into a series of manageable thought-action-observation cycles.

Each iteration of the loop consists of several distinct steps:

  1. Observation: The agent starts with an initial state, which includes the user's high-level goal and any relevant context. In subsequent cycles, the observation is the outcome of the previous action.

  2. Thought: The system presents the goal, the history of previous steps, and the list of available tools to the LLM. It prompts the model to "think" about its next move. This often involves a technique called Chain of Thought, where the model explicitly writes out its reasoning. For example, it might output: "I need to find out the current weather in San Francisco. To do this, I should use the get_weather tool."

  3. Action: Based on its thought process, the LLM selects a tool to use and determines the parameters for it. Modern models are often fine-tuned for "tool use" or "function calling", allowing them to produce a structured JSON[4] object specifying the action, such as { "tool": "get_weather", "parameters": { "city": "San Francisco" } }. This is far more reliable than parsing natural language.

  4. Execution: The agent's orchestration code parses the action specified by the LLM. It then calls the corresponding function or API in the application's backend. The get_weather function is executed with the parameter city="San Francisco".

  5. Update: The result of the tool's execution (e.g., a weather report or an error message) is captured. This result becomes the new observation. The system appends the thought, action, and observation from the completed step to its history and begins the loop again. This cycle continues until the LLM concludes that the original goal has been achieved and decides to take a finish action.

This loop transforms a static LLM into a dynamic problem-solver that can interact with its environment to gather information and accomplish objectives.

The Components of an Agentic System

An agent is a composite system, and its performance depends on the quality of its individual parts. Three primary components define an agent's capabilities: the reasoning engine, the memory system, and the set of tools.

The Reasoning Engine: The LLM

The LLM is the brain of the agent. Its ability to understand instructions, reason about problems, and correctly format its output for tool use is the single most important factor in agent performance. Early attempts at building agents with less capable models often resulted in unreliable systems that would get stuck in loops or fail to follow instructions. The emergence of highly capable models like OpenAI's GPT-4[5], Google's Gemini[6] family, and Anthropic's Claude 3[7] made practical agents viable.

Key characteristics of a good reasoning engine include:

  • Strong Instruction Following: The model must reliably adhere to the complex prompt that structures the thought-action loop.
  • Function Calling/Tool Use: Native support for structured output generation is crucial. It allows the model to specify actions in a machine-readable format, eliminating brittle output parsing.
  • Reasoning Ability: The model needs to break down problems, handle errors reported by tools, and correct its course when an action fails.

Memory: Giving the Agent Context

Memory allows an agent to maintain coherence across the multiple steps of a task. Without it, each turn of the loop would be independent of the last, making it impossible to solve anything but the simplest problems. Agent memory is typically divided into two categories.

Short-Term Memory is the context of the current task. It's essentially a running log of the thoughts, actions, and observations that have occurred so far. This log is included in the prompt sent to the LLM on each iteration of the loop. This is often called a "scratchpad." The primary limitation here is the LLM's context window size. As the agent performs more steps, the history grows, and eventually, it may exceed the model's capacity. The claim that the Claude 3 family offers a 200K context window is outdated. While 200K was the initial context window at launch, Anthropic has since stated that the models are capable of accepting inputs over 1 million tokens for select customers, and some newer models in the family have a 1M token context window. The 200K figure is not the current maximum for the entire family.

Long-Term Memory allows an agent to recall information across different sessions or tasks. This is almost always implemented using an external database, most commonly a vector database[8] like Pinecone[9] or Weaviate[10]. The mechanism works as follows:

  1. When the agent learns a piece of information worth saving, the text is converted into a numerical vector using an embedding model[11].
  2. This vector is stored in the vector database.
  3. During a future task, the agent can form a query based on its current thought process (e.g., "What does my user prefer for project management?").
  4. This query is also converted into a vector, and a similarity search[12] is performed on the database to retrieve the most relevant stored memories.
  5. These retrieved memories are then added to the agent's short-term memory as context for the current step.

Tools: Interacting with the World

Tools are what make an agent useful. They are simply functions that the agent's orchestration code makes available to the LLM. The LLM does not execute the tools itself; it only requests their execution. Each tool must be described to the LLM, including its name, purpose, and required parameters.

For example, a prompt might include a tool definition like this:

{
 "name": "get_stock_price",
 "description": "Retrieves the current price of a stock using its ticker symbol.",
 "parameters": {
 "type": "object",
 "properties": {
 "ticker": {
 "type": "string",
 "description": "The stock ticker symbol, e.g., GOOGL or MSFT."
 }
 },
 "required": ["ticker"]
 }
}

This definition, provided via an API like OpenAI's function calling feature[13], gives the LLM the information it needs to correctly invoke the tool. Common tools for agents include:

  • Web Search: To find up-to-date information.
  • Code Execution: To perform calculations, run scripts, or analyze data. This requires a secure sandbox[14] environment.
  • Database Queries: To retrieve or store structured data.
  • API Calls: To interact with any external or internal service.

A Concrete Example: A Research Agent

Let's walk through a simplified example of an agent tasked with: "Summarize the key findings of the original Transformer paper, 'Attention Is All You Need'."

The agent is equipped with two tools: web_search(query: str) and read_url(url: str). The process is orchestrated by a Python[15] script.

Loop 1

  • Goal: "Summarize 'Attention Is All You Need'."
  • LLM Thought: "I need to find the paper first. I will use the web search tool to find its official page or PDF."
  • LLM Action: {"tool": "web_search", "parameters": {"query": "Attention Is All You Need paper arxiv"}}
  • Execution: The system calls a search API. The top result is the paper's page on arXiv: https://arxiv.org/abs/1706.03762.
  • Observation: "Search results returned a primary link: https://arxiv.org/abs/1706.03762[16]."

Loop 2

  • Goal: "Summarize 'Attention Is All You Need'."
  • Context: (Contains the history from Loop 1).
  • LLM Thought: "I have the link to the paper. Now I need to read its content to summarize it. I will use the read_url tool."
  • LLM Action: {"tool": "read_url", "parameters": {"url": "https://arxiv.org/abs/1706.03762"}}
  • Execution: The system fetches the content from the URL (let's assume it scrapes the abstract and introduction).
  • Observation: "Content from URL: 'The dominant sequence transduction models are based on complex recurrent or convolutional neural networks... We propose a new simple network architecture, the Transformer, based solely on attention mechanisms...'"

Loop 3

  • Goal: "Summarize 'Attention Is All You Need'."
  • Context: (Contains history from Loops 1 and 2).
  • LLM Thought: "I have read the introductory content of the paper. It introduces the Transformer architecture and its reliance on self-attention instead of RNNs. I now have enough information to provide a summary."
  • LLM Action: {"tool": "finish", "parameters": {"summary": "The paper 'Attention Is All You Need' introduced the Transformer, a novel network architecture for sequence transduction tasks that avoids recurrence and instead relies entirely on a self-attention mechanism to process input and output sequences."}}
  • Execution: The system sees the finish action and returns the summary to the user, terminating the loop.

Here is a conceptual code block showing how this could be structured using a library like LangChain[17].

from langchain_openai import ChatOpenAI
from langchain.agents import tool, AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate

# This is a simplified example. Real tools would have proper implementations.
@tool
def web_search(query: str) -> str:
 """Performs a web search for the given query."""
 # In a real app, this would call a search API.
 if "Attention Is All You Need" in query:
 return "Search results: The paper is available at https://arxiv.org/abs/1706.03762"
 return "No results found."

@tool
def read_url(url: str) -> str:
 """Reads the content of a given URL."""
 # In a real app, this would use a library like BeautifulSoup.
 if "1706.03762" in url:
 return "The paper introduces the Transformer architecture, based solely on attention mechanisms..."
 return "Could not read URL."

# Define the tools and the LLM
tools = [web_search, read_url]
llm = ChatOpenAI(model="gpt-4-turbo", temperature=0)

# Create the agent prompt
prompt = ChatPromptTemplate.from_messages([
 ("system", "You are a helpful research assistant."),
 ("human", "{input}"),
 ("placeholder", "{agent_scratchpad}"),
])

# Construct the agent
agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Run the agent
agent_executor.invoke({"input": "Summarize the key findings of the 'Attention Is All You Need' paper."})

The Engineering Reality: Tradeoffs and Failure Modes

While powerful, agentic systems introduce significant engineering challenges related to cost, reliability, and security. They are not a silver bullet.

Cost and Latency

Every step in an agent's loop is at least one LLM API call. A task that takes an agent five steps to complete will incur at least five times the cost and latency of a single, direct LLM call, plus the execution time of the tools themselves. The context (short-term memory) grows with each step, increasing the number of input tokens for subsequent calls. A complex agent can quickly become expensive to run. For example, using a premium model like GPT-4 Turbo, a 5-step agent with an average of 4000 tokens of context per step could cost significantly more than a single query, as detailed on the OpenAI pricing page[18].

Reliability and Determinism

LLMs are probabilistic. An agent given the same goal twice may choose a different path each time. This non-determinism makes debugging and quality assurance difficult. Common failure modes include:

  • Hallucination: The agent may try to call a tool that doesn't exist, or it might invent parameters for a tool.
  • Looping: An agent can get stuck in a repetitive cycle. For example, it might repeatedly search for information it already found because it fails to correctly parse the observation.
  • Poor Error Handling: If a tool returns an error (e.g., an API is down), a less capable model might not understand how to recover and may give up or repeat the failing action.

These issues plagued early agent experiments like Auto-GPT[19] and BabyAGI[20], which were impressive demos but often failed at completing non-trivial tasks reliably.

Security

Giving an LLM access to tools is a serious security consideration. An agent with the ability to execute code, write to files, or access internal APIs is a powerful capability that can be misused. The most significant threat is Indirect Prompt Injection. If an agent reads a malicious instruction from an external data source (like a webpage or a document), it might be tricked into executing a harmful command on behalf of an attacker, such as deleting a database or sending confidential data to an external server.

Mitigating this risk requires strict controls. Any tool that executes code must do so in a tightly controlled and ephemeral sandbox environment. For sensitive actions, such as sending an email or modifying a production resource, a human-in-the-loop approval step is essential.

The Decision Lens for Technical Leaders

Deciding whether to build an agent requires a clear-eyed assessment of the problem and the technology's current limitations.

When to Build an Agent

Agentic architectures are best suited for problems that are dynamic and require multiple steps of reasoning and interaction, where the exact path to a solution cannot be scripted in advance. Good use cases include:

  • Automated Scientific Research: An agent can read papers, run code to analyze data, and generate hypotheses.
  • Complex Data Analysis: An agent can be given access to a database and data analysis tools to answer high-level business questions.
  • Dynamic DevOps Tasks: An agent could monitor a system, diagnose an issue by querying logs and metrics, and attempt a remediation by calling a cloud provider's API.

When NOT to Build an Agent

Do not use an agent for tasks that are simple, well-defined, or require high reliability and low latency. For many problems, a simpler architecture is better.

  • Information Retrieval: If the goal is to answer questions over a set of documents, a standard Retrieval-Augmented Generation (RAG)[21] system is cheaper, faster, and more reliable than an agent.
  • Deterministic Workflows: If a task follows a predictable, scriptable sequence of steps, use traditional code. An agent adds unnecessary cost, latency, and unpredictability.
  • High-Stakes Environments: For mission-critical applications where failure is not an option, the probabilistic nature of today's agents makes them a poor choice for fully autonomous operation.

Frameworks like LangChain[22] and LlamaIndex[23] provide helpful abstractions for building agents, but they also introduce their own complexity. Building the core loop from scratch with direct API calls offers maximum transparency and control, which can be preferable for production systems.

The State of AI Agents Today

The field of AI agents is maturing rapidly. The initial wave of excitement in 2023 was driven by open-source projects that showed what was possible, but also highlighted the severe reliability issues. Today, the focus has shifted toward building more constrained, reliable, and practical agents.

Progress is primarily driven by improvements in the underlying LLMs. Larger context windows and more accurate function calling capabilities directly translate to more capable agents. The technology is still in the early adoption phase for production use cases. Most successful deployments are either internal tools for expert users or systems with a human-in-the-loop to supervise and approve actions.

It is critical to correct the common misconception that these agents are "thinking" or possess general intelligence. An AI agent is a software system executing a loop. Its apparent intelligence is an emergent property of a powerful LLM's pattern-matching ability combined with well-defined tools and a clear objective. The magic is not in a single component, but in the architecture that brings them together.

Sources (23)
  1. Large Language Model (Wikipedia)
  2. API (Wikipedia)
  3. ReAct: Synergizing Reasoning and Acting in Language Models (arXiv)
  4. JSON Standard
  5. GPT-4 Research
  6. Google Gemini
  7. Anthropic Claude 3 Family
  8. Vector database (Wikipedia)
  9. Pinecone
  10. Weaviate
  11. Word embedding (Wikipedia)
  12. Nearest neighbor search (Wikipedia)
  13. OpenAI Function Calling Guide
  14. Sandbox (computer security) (Wikipedia)
  15. Python Official Website
  16. Attention Is All You Need (arXiv)
  17. LangChain
  18. OpenAI Pricing
  19. Auto-GPT (GitHub)
  20. BabyAGI (GitHub)
  21. Retrieval-Augmented Generation (IBM Research)
  22. LangChain Python Docs
  23. LlamaIndex

Recommended reading

Function Calling and Tool Use in LLMs14 min readA comprehensive guide to how LLMs use functions and tools to interact with external systems, covering the underlying mechanism, history, practical examples, and critical decision-making tradeoffs for technical leaders.Evaluating LLMs: How to Measure Output Quality13 min readA comprehensive guide for technical leaders on measuring LLM performance, covering human evaluation, automated metrics like BLEU and BERTScore, and the modern LLM-as-a-Judge approach.The History of the Chatbot: From ELIZA to ChatGPT12 min readA comprehensive reference history of conversational AI, tracing its evolution from the simple pattern-matching programs of the 1960s to the large language models of the present day.Semantic Caching: A Technical Guide for LLM Applications11 min readSemantic caching reduces LLM application costs and latency by storing and reusing responses to conceptually similar queries, using vector embeddings and similarity search instead of exact-text matching.Speculative Decoding: How LLMs Generate Faster12 min readSpeculative decoding accelerates large language model inference by using a smaller, faster model to draft text that a larger, more powerful model then verifies in parallel, significantly reducing generation latency without compromising output quality.LLM Observability: Tracing, Logging, and Cost Tracking11 min readA comprehensive guide to LLM observability, explaining the technical mechanisms of tracing, logging, evaluation, and cost analysis for production AI systems.