LLM Agents and Tool Use

TL;DR

Intuition: agents are planners + tool users + memory

Strip away the framework names and an LLM agent is three pieces glued together. The planner is the LLM itself, queried with the current state and asked to emit the next step in some structured format. The tool surface is the set of side-effecting actions the harness exposes (a search API, a Python sandbox, a filesystem read, a browser click). The memory is whatever the harness puts back into the prompt on the next turn: the conversation so far, retrieved documents, summarized scratchpads, persistent vector stores. The loop is prompt -> emit action -> execute -> observe -> repeat, and every interesting design choice is about what gets carried from one turn to the next.

That decomposition matters because each piece fails differently. A planner failure looks like a bad next step on a clean state; the fix is better prompting, more capable model, or fine-tuning on better trajectories. A tool-surface failure looks like a correct plan that the harness cannot execute; the fix is more tools, better schemas, or a sandbox that returns useful errors. A memory failure looks like the agent forgetting the goal twenty turns in, repeating itself, or cycling through the same three actions; the fix is summarization, observation truncation, and a hard turn cap. The frameworks below (LangChain, AutoGen, LangGraph, CrewAI) are all opinionated takes on how to wire these three pieces together. None of them removes the underlying decomposition; pick a framework that surfaces it cleanly so you can debug it.

1. 2022Chain-of-thought promptingpaper

   Wei et al. show that adding "Let us think step by step" or a few worked examples of reasoning to the prompt unlocks multi-step arithmetic, commonsense, and symbolic tasks. CoT is the planner half of every agent that came after.

2. 2022ReAct: Reason + Actpaper

   Yao et al. interleave free-form reasoning ("Thought:") with discrete tool calls ("Action:") and ground each step in tool output ("Observation:"). On HotpotQA and ALFWorld, ReAct beats both CoT-only and act-only baselines because reasoning can route and observations can correct.

3. 2023Toolformer: self-supervised tool-usepaper

   Schick et al. teach a base LM to use a calculator, search engine, calendar, translator, and Q&A by sampling candidate API calls and keeping the ones that reduce next-token perplexity. The model fine-tunes itself on the keepers.

4. 2023Tree of Thoughtspaper

   Yao et al. lift CoT from a single chain to a search tree: each node is a partial solution, an evaluator (also an LLM call) prunes branches, and BFS or DFS finds the best leaf. ToT wins on Game-of-24, mini-crosswords, and creative writing where deliberate search beats greedy decoding.

5. 2023Reflexion: verbal reinforcementpaper

   Shinn et al. add a self-critique step after each failed attempt. The model writes a short "I should have ..." note that gets appended to the prompt for the next attempt. Cheap, no gradient updates, and lifts pass rates on coding and decision-making benchmarks.

6. 2023AutoGen and the multi-agent waveframework

   Wu et al. ship a multi-agent conversation framework. AutoGPT, BabyAGI, MetaGPT, CrewAI, and LangGraph follow within months. Most production agents in 2024 still look like a single ReAct loop with tools; the multi-agent wins are real but specific.

ReAct: the Reason + Act loop

The single most-copied agent recipe in 2024 is ReAct [2]. It is a prompt template, not a model, and the template is simple enough to reproduce in a few lines.

Question: Which film came out first, Inception or Interstellar?

Thought: I need the release years of Inception and Interstellar.
Action: Search[Inception movie release year]
Observation: Inception is a 2010 science fiction film directed by Christopher Nolan.

Thought: Inception is 2010. Now I need Interstellar's release year.
Action: Search[Interstellar movie release year]
Observation: Interstellar is a 2014 science fiction film directed by Christopher Nolan.

Thought: 2010 is earlier than 2014, so Inception came out first.
Action: Finish[Inception]

ReAct is load-bearing, not flashy. Most production agents (LangChain agents, OpenAI Assistants, the original AutoGPT inner loop) are ReAct or a thin variant, and most “advanced” agent frameworks are also ReAct underneath with a different parser. If your agent is broken and you do not know why, instrument the trace and read it: the bug will be in a Thought, an Action argument, or an Observation that does not match what was asked for.

A maximum-likelihood loss on ReAct trajectories

When you fine-tune a model on ReAct trajectories, the loss is just next-token cross-entropy on the trace, with one important nuance: you should not include the observation tokens in the loss, because the model does not generate observations. Let a trajectory be the sequence

where $x$ is the user task, $t_i$ is the $i$-th thought (tokens), $a_i$ is the action token sequence, and $o_i$ is the observation returned by the harness. The loss masks out $x$ and every $o_i$:

where $\mathrm{prefix}_{<u}$ denotes every token in $\tau$ that comes before $u$ in linear order. The mask is the only difference from standard SFT and is the difference between an agent that knows when to act and one that hallucinates observation tokens itself [7].

Toolformer: self-supervised tool-use

Toolformer [3] answers a different question: how do you teach a base LM to use tools without expensive labelled trajectories? The trick is fully self-supervised.

Start with a corpus of plain text and a small set of tools (calculator, Wikipedia search, calendar, translator, question answerer). For every position in the corpus, the base LM is prompted with a few-shot examples and asked to propose an API call that would fit at that position, with its argument. Each proposal is then executed to get the response. For each candidate API-call-and-response, compare the next-token perplexity of the original continuation with the API call inserted versus without. Keep only the calls where the perplexity drop is at least a threshold $\tau$ on the actual next tokens.

The result is an automatically labelled dataset of (text, tool-call, response, kept-or-not) tuples; fine-tune the base LM on the kept tuples. The fine-tuned model now emits <API> tool(args) -> result </API> tokens in line, and at inference time the harness intercepts those tokens, executes the call, and replaces the result with the real one before continuing generation.

Toolformer’s contribution is methodological more than practical (the original model is a 6.7B GPT-J, and modern instruction-tuned LLMs already use tools competently). The transferable idea is scoring tool calls by how much they reduce perplexity on the actual next tokens, which gives a clean self-supervised signal whenever you can re-run the LM on the post-tool-call context.

Function calling and tool-use APIs

Production agents do not parse Action: Search[query] strings. They use structured function calling: the model emits a JSON object with a tool name and arguments, validated against a JSON Schema, and the harness dispatches accordingly.

Three flavours dominate in 2024.

OpenAI function calling. Tools are declared with a JSON Schema (name, description, parameters). The model emits a tool_calls array on its message; the harness runs each tool, appends the results as messages with role: "tool", and queries the model again. The schema is enforced by the model server, so malformed JSON is rare in practice.

Anthropic tool use. Same shape (declare tools with JSON Schema, model emits a tool_use content block, harness appends tool_result blocks). Anthropic also exposes a separate computer use modality where the tool surface is the screen itself: actions like screenshot, mouse_move, mouse_click, key, type. The model receives observations as image tokens.

Google function calling on Gemini. Same idea, slightly different wire format; supports parallel function calls in a single turn.

The portable lesson is that the schema is the contract. A tool with a vague description and loose argument types invites the model to invent calls; a tool with a tight schema and a clear description gets called correctly. Write the schema like API documentation, not like an internal helper.

Tree of Thoughts: deliberate search

ReAct is a greedy left-to-right walk. For tasks where the next correct step is not obvious, search beats greedy. Tree of Thoughts [4] formalizes that idea.

The lesson is sharper than “search beats greedy”. CoT is implicitly a single path drawn by the model’s sampling distribution; ToT lets you draw many paths and pick the best. If your task has a clear evaluator (does this code compile, does this proof type-check, is this state closer to the goal) ToT’s evaluator step has real signal and search pays off. If your evaluator is itself a vague LLM judge, ToT mostly adds cost.

Reflexion: verbal reinforcement

Reflexion [5] is the cheapest agent-improvement trick in the literature: no gradients, no reward model, no extra fine-tune. After each failed attempt, the model writes a short verbal reflection (“I tried $X$, that failed because $Y$, next time I should $Z$”), the reflection gets appended to the agent’s long-term memory, and the next attempt sees both the original task and the accumulated reflections.

Three observations:

Verbal feedback works because LLMs are good in-context learners. A reflection is just more context. Telling the model “last time you forgot to check the date, do not forget this turn” is functionally similar to a reinforcement-learning update, except the update lives in the prompt rather than in the weights.

It needs a real failure signal. Reflexion needs an external “you failed” signal: a unit test, a benchmark grader, an environment terminal reward. Without that signal you cannot tell which attempt to reflect on, and the reflection becomes performative.

It interacts cleanly with ReAct. A common production setup is ReAct as the inner loop, Reflexion as the outer loop: each ReAct trajectory either finishes or hits a turn cap, the outer loop checks whether the answer passes, and on failure produces a one-paragraph reflection that goes into the system prompt for the next attempt. Reflexion lifts HumanEval pass-rates by several points and AlfWorld success rates by tens of percentage points [5].

The pattern is general: store the lesson, not the weights. For tasks with cheap external graders, this is one of the highest-leverage agent tricks.

Multi-agent systems

Once you have a single agent that works, the natural next step is multiple agents talking to each other. Three production frameworks dominate.

AutoGen [6]. Agents are first-class objects with names, system prompts, and tools. They send messages to each other through a conversation manager. The headline patterns are user proxy + assistant (the user proxy executes code that the assistant writes) and group chat (an explicit speaker-selection rule among $N$ agents). AutoGen is closest to “ReAct generalized to multi-party conversation”; the bookkeeping is messages and turn order rather than Thought / Action / Observation.

CrewAI. Agents have roles and goals, and tasks are explicitly delegated by a manager agent. The mental model is a small fixed org chart: a researcher, a writer, a critic. Useful when you can sketch the org chart up front; less useful when the role boundaries are fluid.

LangGraph. Drops the conversational metaphor entirely and exposes the agent as a directed graph of nodes (LLM calls, tool calls, conditional branches). This is the most explicit framework, and the easiest one in which to build agents that are not loops but actual state machines. LangGraph is also the framework most often used for production agents in 2024 because it makes the control flow visible.

The honest summary is that multi-agent wins are real but specific. They show up cleanly when the task has a clear separation of responsibilities (research vs writing vs critique), where one agent has a system prompt the other should not see, or where you want explicit role-level safety boundaries (a separate agent that has the database write tool and refuses on unsafe inputs). For a single end-to-end task with no role separation, a single ReAct agent with the right tools usually beats a multi-agent system on both quality and latency.

Plan-and-execute architectures

Plan-and-execute is the third common control structure. It splits the loop into two phases.

Phase 1: plan. A planner LLM produces an explicit step-by-step plan (often as a numbered list or a JSON DAG) for the whole task at once.

Phase 2: execute. A separate executor LLM (or the same LLM with a different prompt) walks the plan one step at a time, calling tools, and appending results.

Compared to ReAct, plan-and-execute has two strengths and one weakness. The strengths: the plan is reviewable up front (a human or a separate critic can check it before any tool is called), and the executor’s per-step prompt is shorter (it only needs the current step plus relevant context, not the whole trace). The weakness: a plan made up front cannot use information that only appears mid-execution, so plan-and-execute is brittle on tasks where early observations should change the plan. Hybrid forms (“plan, execute one step, optionally re-plan”) restore the adaptivity at the cost of more LLM calls and a more complex harness.

Memory: short-term, long-term, and the limits of the context window

The context window is the agent’s short-term memory. Everything in the current prompt (system, task, trace so far, retrieved chunks) is “remembered”; everything outside is forgotten unless the harness re-injects it. Long-context models (Claude $200$k, Gemini $1$M, GPT-4 turbo $128$k) push that horizon out, but they do not change the structural problem: bigger context is more expensive per token, slower to attend over, and increasingly noisy as you stuff it.

Production agents handle memory in three layers.

Short-term: the live trace. The current ReAct trace, with observations truncated to a per-tool budget. A naive harness lets a single observation balloon to $20$k tokens (a full HTML page, a verbose stack trace, a giant JSON dump) and then the next prompt has no room for further reasoning. The fix is a per-tool truncation policy: top-$k$ snippets, head-and-tail truncation, or a quick summarization call.

Medium-term: summarized scratchpad. Once the trace passes a threshold, the harness summarizes the older half (“so far the agent has tried $A$, $B$, $C$ and learned that $X$ does not work because $Y$”) and replaces it with the summary. The summarization step is one extra LLM call per trim and trades fidelity for room.

Long-term: vector store. Cross-task memories (a coding agent remembering past projects, a research agent remembering past queries) live in a vector store and are retrieved like any other RAG corpus. The retrieved snippets are appended to the current prompt as part of the system message. This is where agent memory blurs into RAG; the difference is mostly whether the corpus is the user’s documents (RAG) or the agent’s past traces (long-term agent memory).

Computer use, browser use, and code-execution sandboxes

Two parallel evolutions extended the agent surface in 2024. Both are about what the agent can do in the world.

Computer use. Anthropic’s Claude computer use and OpenAI’s Operator expose the desktop or browser as a tool. The action space includes screenshot, mouse_move, mouse_click, key, and type; the observation is an image of the screen. The model has to read pixels to know where to click, which means it has to be a strong vision model. Claude 3.5 Sonnet (computer use) and the later Claude 3.7 / 4 generations are the practical state of the art; pass rates on OSWorld and similar benchmarks are still in the $20$ to $40\%$ range as of late 2024.

Browser use. A narrower version of computer use: the harness exposes navigate, click(selector), fill(selector, text), extract, screenshot. The DOM is a much cleaner state than the screen, so the agent can run on cheaper text-only models. Browser-Use, Stagehand, and Playwright-based wrappers are the common implementations; AgentBench and WebArena are the evaluation benchmarks. Pass rates here are higher than on full computer use (roughly $30$ to $60\%$ depending on task) because the abstraction is cleaner.

Code execution sandboxes. E2B and Riza (and the older Code Interpreter / Advanced Data Analysis surface) wrap a Python or Node sandbox as a tool. The harness ships a python(code) action; the sandbox runs the code, captures stdout / stderr, and returns the result. This is the highest-signal tool for an agent because the failure mode is unambiguous (the code either works or raises) and the action space (write a Python program) is enormous. Code-execution sandboxes are the closest production agents get to real generality; most non-trivial agent benchmarks (GAIA, SWE-Bench) require some form of code execution.

A custom ReAct loop in fifty lines

A useful reality check is that the entire ReAct pattern fits in a small file. Here is a self-contained loop in pure Python plus the OpenAI client, alongside the equivalent in three production frameworks.

import re
from openai import OpenAI

client = OpenAI()

def search(q):
  return f"(stub) results for: {q}"

def calc(expr):
  return str(eval(expr, {"__builtins__": {}}, {}))

TOOLS = {"Search": search, "Calc": calc}

SYSTEM = """You are a ReAct agent. On each turn emit:
Thought: <one sentence reasoning>
Action: <ToolName>[<arg>]
or, when you have the answer:
Action: Finish[<answer>]
Stop after the Action line."""

def run(task, max_turns=8):
  trace = f"Question: {task}\n"
  for _ in range(max_turns):
      msg = [
          {"role": "system", "content": SYSTEM},
          {"role": "user", "content": trace + "Thought:"},
      ]
      out = client.chat.completions.create(
          model="gpt-4o-mini",
          messages=msg,
          stop=["Observation:"],
          temperature=0,
      ).choices[0].message.content
      trace += "Thought:" + out
      m = re.search(r"Action:\s*(\w+)\[(.*?)\]", out, re.S)
      if not m:
          return "(parse error) " + out
      tool, arg = m.group(1), m.group(2)
      if tool == "Finish":
          return arg
      if tool not in TOOLS:
          obs = f"unknown tool: {tool}"
      else:
          obs = TOOLS[tool](arg)
      trace += f"\nObservation: {obs}\n"
  return "(turn cap reached)"

print(run("What is the population of Tokyo, in millions, divided by 2?"))

The custom-loop and framework versions land in roughly the same place semantically. The framework adds tracing, retries, and a more battle-tested parser; the custom loop is fifty lines you can read end to end, which is sometimes the thing you need.

Pitfalls

Pop quiz

Interview Q&A