Code Lab 01 - Agent Loop from Scratch

Write the agent loop yourself - tool calls, argument validation, error results, bounds - point it at a small retail environment, and evaluate it the way agent benchmarks do: on the final database state, over repeated trials, with pass^k. Then vary the prompt and tool descriptions and see what actually moves the numbers.

← Back to Overview: Agent Foundations · Concepts: The Agent Loop · Tool Use · Agent Evaluation Basics

What's In This Lab

flowchart LR
    T["📋 Task"] --> L["🔁 Loop<br/>(agent_lab.py)"]
    L <-->|"chat.completions + tools"| M["🧠 Model server<br/>(any OpenAI-compatible)"]
    L <-->|"validated calls"| E["🏪 Shop environment"]
    E --> G["✅ Grade final DB state<br/>(+ reply regex)"]
    G --> R["📊 pass@1, pass^k, pass@k<br/>per-task successes"]

    style L fill:#e8e0d4,stroke:#c8b89a
    style E fill:#dde4dc,stroke:#b0c4b0
    style G fill:#d8dfe8,stroke:#b0bac8

Run It

Start any OpenAI-compatible server with a tool-calling model:

# Apple silicon
pip install mlx-lm && mlx_lm.server --model mlx-community/Qwen3-8B-4bit          # http://localhost:8080/v1
# NVIDIA GPU
vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes      # http://localhost:8000/v1
# Ollama:  ollama pull qwen3:8b   → base URL http://localhost:11434/v1, model qwen3:8b

Then:

cd 13-Agent-Foundations/CodeLabs/01-Agent-Loop-from-Scratch
pip install -r requirements.txt

python agent_lab.py --no-thinking --limit 3 --trials 1          # smoke test
python agent_lab.py --no-thinking --show-failures               # 18 tasks x 4 trials x 3 variants
python agent_lab.py --base-url https://api.openai.com/v1 --model <model>   # a hosted API (set OPENAI_API_KEY)

--no-thinking sends chat_template_kwargs={"enable_thinking": false}, which Qwen3 on vLLM and mlx-lm understand; leave it off for hosted APIs. Sampling defaults to temperature 1.0, top_p 1.0 so repeated trials can differ.

A reference run (Qwen3-8B 4-bit, thinking off, 4 trials per task):

variant          pass@1    pass^2    pass^4    pass@4  calls/ep    err/ep    tok/ep    sec/ep
full              0.583     0.565     0.556     0.611       2.6       0.1    3971.2       6.4
terse_tools       0.611     0.565     0.556     0.722       2.8       0.3    3208.0       6.9
no_policy         0.708     0.694     0.667     0.722       2.8       0.2    3726.4       7.5

Walkthrough - What to Look At

1. The should-refuse tasks pass in every variant - even with no policy in the prompt. Cancelling a shipped or delivered order fails because cancel_order rejects it and returns an informative error; the model reads the error and explains it to the customer. Rules enforced in tools don't depend on the prompt.
2. The one rule the tools don't enforce is broken in every variant. In other_customer, Eli asks to cancel an order that belongs to Chen. The model skips identification and cancels it - 0/4 with the policy in the prompt, 0/4 without. Prompt rules are advice; code rules are enforcement (Anatomy of an AI Agent).
3. Claimed actions are the largest failure class. In refund_one and refund_shoes the agent looks up the order and then replies "the refund has been processed" without ever calling refund_item; in mixed it refuses the invalid half and forgets to do the valid half. Only a state check catches this - an LLM judge reading the replies would score them as successes.
4. The detailed policy prompt did not help this model - it hurt. no_policy scored highest. The differences come from whole tasks flipping (mixed 0/4 → 4/4) rather than from noise within tasks: pass^4 is close to pass@1 in every variant, so this model is quite consistent for a given prompt, and a prompt change moves entire tasks. With 18 tasks, a difference of two tasks is not strong evidence - which is the point of the next exercise.
5. Terse descriptions cost reliability of calls, not success. Invalid calls rose from 0.08 to 0.32 per episode, but informative error results let the model recover, so success held. Good errors are a safety net for poor documentation - not a substitute for it.
6. Other failure classes: wrong target (already_cancelled cancels Dana's other, pending order), and an unsupported answer (count_items reports "15 items" without reading any order).

Check Yourself

Exercises

References

• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) - state-based grading and pass^k
• Anthropic, Demystifying evals for AI agents (2026)
• Qwen3 function calling and mlx-lm documentation
• vLLM tool calling documentation

Last reviewed: 2026-09