Skip to the content

See what happened inside one answer

An agent answers in steps: a model plans, hands work to helpers, the helpers call tools, and the answer comes back. Pick two recorded runs of the same question and see where the time and the tokens went, and which step went wrong.

  • Recorded runs, not live
  • No model, no key
  • A trace you paste is not sent anywhere

The run behind these traces

One real run from the book AI Engineering, Measured. An orchestrator model hands questions to two worker models: one looks up stock and prices, one searches operations documents. The same six questions were asked many times while two settings changed.

  • Worker step cap: how many model calls a worker may make before it must stop.
  • What happens when a worker reaches its cap. Honest: the worker says it did not finish. Silent: its last words go back as if they were the answer.

Model recorded in the trace: claude-opus-4-6

The run was recorded in English: its questions and the models' words are shown as recorded.

1. Pick two runs

Try a comparison

The question askedWe are short on mice. What would it cost to bring them to 10 units, and what is the change process for ordering hardware?

The question askedWe are short on mice. What would it cost to bring them to 10 units, and what is the change process for ordering hardware?

Look at your own trace

Paste one JSON object per line. It is read inside this page: nothing is sent anywhere, and nothing is kept when you leave.

What a line can hold

A line needs span_id. It may also have parent_span_id, session_id, name, seconds or duration_ms, input_tokens, output_tokens, stop_reason, truncated, error, tool_calls and text. The book's handoff and final records are read too. Lines are grouped by session_id.

One line from the recorded run, as an example:

{"run_id": "2026-09-05T21:46:23+00:00", "kind": "span", "span_id": "s0001", "parent_span_id": null, "session_id": "2026-09-05T21:46:23+00:00|cap2|honest|r1|Q1", "prompt_id": 1, "handler": "honest", "worker_max_steps": 2, "rep": 1, "loop": "orchestrator", "step": 0, "input_tokens": 684, "usage_input_tokens": 684, "usage_output_tokens": 62, "n_messages": 1, "tools_available": ["ask_inventory", "ask_docs"], "tool_calls": [{"name": "ask_inventory", "input": {"task": "How many headsets are currently in stock?"}}], "stop_reason": "tool_use", "text": "", "truncated": false, "seconds": 2.513}

2. What changed between the two runs

Every number is counted from the two traces on this page.

MeasureRun ARun BB minus A
Model calls97-2
Model time, summed (seconds)39.22229.779-9.443
Tokens read8,8106,854-1,956
Tokens written1,4151,268-147
Calls cut off by the step cap20-2
Worker answers reported missing20-2
Unfinished answers passed on as finished000

Where the model time went

Each bar is one run, split by the loop that made the calls. The bars share one scale.

  • Orchestrator

    ARun A: 16.814 s, 3 calls
    BRun B: 11.198 s, 2 calls
  • inventory worker

    ARun A: 11.414 s, 4 calls
    BRun B: 7.35 s, 3 calls
  • docs worker

    ARun A: 10.994 s, 2 calls
    BRun B: 11.231 s, 2 calls

3. Step by step

One line per model call, in the order they happened. Indented calls were made by a worker for the call above. Bars share one scale across both runs.

Run AQ4 · cap 2 · honest · repeat 1

  1. Orchestrator, call 1Asked for ask_inventory, ask_docs

    Time3.644 sTokens705 read, 145 written
    What happened, and how it is measured
    What the model wrote
    Let me look into both of those questions for you simultaneously.
    Tools it asked for
    • ask_inventory {"task":"How many mice are currently in stock, what is the price per unit, and how much would it cost to bring the stock level up to 10 units?"}
    • ask_docs {"task":"What is the change process or procedure for ordering hardware?"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 23 of the trace

  2. inventory worker, call 1Asked for lookup_stock

    Time2.363 sTokens723 read, 66 written
    What happened, and how it is measured
    What the model wrote
    Let me look up the stock information for mice first.
    Tools it asked for
    • lookup_stock {"query":"mouse"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 24 of the trace

  3. inventory worker, call 2Cut off by the capAsked for calculate

    Time3.175 sTokens834 read, 110 written
    What happened, and how it is measured
    What the model wrote
    Now I know the current stock is **0 units** and the price is **$19.50** per unit. Since we need to bring the stock level up to 10, that means purchasing 10 units. Let me calculate the cost.
    Tools it asked for
    • calculate {"expression":"10 * 19.5"}
    Why it stopped
    This was the last call the cap allowed. The model was still asking for a tool, so the work stopped here.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.Cut off: the loop writes this into the record when it stops a worker at its cap. It is recorded when it happens, not guessed later.

    Line 25 of the trace

  4. Back to the orchestrator: reported as not finished
    What the worker passed back
    Nothing.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.

    Line 26 of the trace

  5. docs worker, call 1Asked for search_docs, search_docs

    Time2.574 sTokens620 read, 112 written
    What happened, and how it is measured
    What the model wrote
    Let me search for information about the change process for ordering hardware.
    Tools it asked for
    • search_docs {"question":"change process or procedure for ordering hardware"}
    • search_docs {"question":"hardware ordering procedure"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 27 of the trace

  6. docs worker, call 2Answered

    Time8.42 sTokens1,571 read, 332 written
    What happened, and how it is measured
    What the model wrote
    The search results contain information about **change management windows** and **hardware replacement procedures**, but they do not contain any specific procedure or process for **ordering hardware** (e.g., procurement, purchase orders, vendor ordering, or supply chain processes). Here is what the documents **do** cover that may be tangentially relevant: 1. **Change window classifications** for hardware-related changes (like line card replacements), which are based on device tier: > *"Tier 1 devices are core routers and core switches. Tier 2 devices are edge routers, aggregation switches, and route reflectors. Tier 3 devices are access switches, out-of-band console servers, and lab equipment. Changes to Tier 1 devices require a four-hour approved window, submitted no later than ten business days before the change date. The window must be approved by the change advisory board and by the on-call network lead for the region. Changes to Tier 2 devices require a two-hour window submitted five business days ahead. Tier 3 changes may be made during the standing weekly window with no separate submission."* 2. **Hardware replacement procedures** (e.g., MOP-021 — Line Card Replacement) and **chassis inventory states** (REF-050), which deal with the operational side of swapping hardware. However, **the documents do not contain a procedure specifically for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
    Why it stopped
    It finished and gave its answer.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 28 of the trace

  7. Back to the orchestrator: an answer
    What the worker passed back
    The search results contain information about **change management windows** and **hardware replacement procedures**, but they do not contain any specific procedure or process for **ordering hardware** (e.g., procurement, purchase orders, vendor ordering, or supply chain processes). Here is what the documents **do** cover that may be tangentially relevant: 1. **Change window classifications** for hardware-related changes (like line card replacements), which are based on device tier: > *"Tier 1 devices are core routers and core switches. Tier 2 devices are edge routers, aggregation switches, and route reflectors. Tier 3 devices are access switches, out-of-band console servers, and lab equipment. Changes to Tier 1 devices require a four-hour approved window, submitted no later than ten business days before the change date. The window must be approved by the change advisory board and by the on-call network lead for the region. Changes to Tier 2 devices require a two-hour window submitted five business days ahead. Tier 3 changes may be made during the standing weekly window with no separate submission."* 2. **Hardware replacement procedures** (e.g., MOP-021 — Line Card Replacement) and **chassis inventory states** (REF-050), which deal with the operational side of swapping hardware. However, **the documents do not contain a procedure specifically for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.

    Line 29 of the trace

  8. Orchestrator, call 2Asked for ask_inventory

    Time3.425 sTokens1,312 read, 102 written
    What happened, and how it is measured
    What the model wrote
    Let me retry the inventory question.
    Tools it asked for
    • ask_inventory {"task":"What is the current stock level of mice and what is the price per unit for a mouse? I need to know how much it would cost to purchase enough mice to bring the total stock to 10 units."}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 30 of the trace

  9. inventory worker, call 1Asked for lookup_stock

    Time3.036 sTokens733 read, 67 written
    What happened, and how it is measured
    What the model wrote
    Let me start by looking up the stock information for mice.
    Tools it asked for
    • lookup_stock {"query":"mouse"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 31 of the trace

  10. inventory worker, call 2Cut off by the capAsked for calculate

    Time2.84 sTokens845 read, 125 written
    What happened, and how it is measured
    What the model wrote
    Here are the current details for the mouse (SKU: MS-02): - **Current Stock Level:** 0 units - **Price Per Unit:** $19.50 To bring the stock to **10 units**, you'd need to purchase **10 mice**. Let me calculate the total cost:
    Tools it asked for
    • calculate {"expression":"10 * 19.5"}
    Why it stopped
    This was the last call the cap allowed. The model was still asking for a tool, so the work stopped here.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.Cut off: the loop writes this into the record when it stops a worker at its cap. It is recorded when it happens, not guessed later.

    Line 32 of the trace

  11. Back to the orchestrator: reported as not finished
    What the worker passed back
    Nothing.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.

    Line 33 of the trace

  12. Orchestrator, call 3Answered

    Time9.745 sTokens1,467 read, 356 written
    What happened, and how it is measured
    What the model wrote
    Here's what I was able to find: --- ### 🖱️ Mice Inventory & Cost Unfortunately, the inventory system wasn't able to return the current stock level or pricing for mice at this time. You may need to check the inventory system directly or contact the procurement team for current stock counts and unit pricing. --- ### 📋 Change Process for Ordering Hardware The documentation doesn't outline a specific **procurement/ordering** procedure for hardware. However, it does cover the **change management process** that applies when hardware changes are involved. Here's a summary: | Device Tier | Examples | Change Window | Lead Time | Approval | |---|---|---|---|---| | **Tier 1** | Core routers & core switches | 4-hour approved window | 10 business days | Change Advisory Board + on-call network lead | | **Tier 2** | Edge routers, aggregation switches, route reflectors | 2-hour window | 5 business days | Standard submission | | **Tier 3** | Access switches, console servers, lab equipment | Standing weekly window | No separate submission required | — | Mice would likely fall under **Tier 3 (or general lab/access equipment)**, meaning changes can be made during the standing weekly window with no separate submission needed. > **Note:** For the actual **purchasing/procurement process** (e.g., purchase orders, vendor ordering), you may need to consult a separate procurement or supply chain policy, as it wasn't covered in the network operations documentation. Would you like me to try looking up anything else?
    Why it stopped
    It finished and gave its answer.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 34 of the trace

Run BQ4 · cap 5 · honest · repeat 1

  1. Orchestrator, call 1Asked for ask_inventory, ask_docs

    Time3.844 sTokens705 read, 145 written
    What happened, and how it is measured
    What the model wrote
    Let me look into both of those questions for you simultaneously.
    Tools it asked for
    • ask_inventory {"task":"How many mice are currently in stock, what is the price per unit, and how much would it cost to bring the stock level up to 10 units?"}
    • ask_docs {"task":"What is the change process or procedure for ordering hardware?"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 324 of the trace

  2. inventory worker, call 1Asked for lookup_stock

    Time2.485 sTokens723 read, 67 written
    What happened, and how it is measured
    What the model wrote
    Let me start by looking up the stock information for mice.
    Tools it asked for
    • lookup_stock {"query":"mouse"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 325 of the trace

  3. inventory worker, call 2Asked for calculate

    Time2.246 sTokens835 read, 76 written
    What happened, and how it is measured
    What the model wrote
    Now let me calculate the cost to bring the stock level up to 10 units.
    Tools it asked for
    • calculate {"expression":"10 * 19.5"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 326 of the trace

  4. inventory worker, call 3Answered

    Time2.619 sTokens926 read, 76 written
    What happened, and how it is measured
    What the model wrote
    Here's the full breakdown: - **Product:** Mouse (SKU: MS-02) - **Current stock:** 0 units - **Price per unit:** $19.50 - **Units needed to reach 10:** 10 - **Total cost to restock to 10 units:** **$195.00**
    Why it stopped
    It finished and gave its answer.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 327 of the trace

  5. Back to the orchestrator: an answer
    What the worker passed back
    Here's the full breakdown: - **Product:** Mouse (SKU: MS-02) - **Current stock:** 0 units - **Price per unit:** $19.50 - **Units needed to reach 10:** 10 - **Total cost to restock to 10 units:** **$195.00**
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.

    Line 328 of the trace

  6. docs worker, call 1Asked for search_docs, search_docs

    Time2.802 sTokens620 read, 116 written
    What happened, and how it is measured
    What the model wrote
    Let me search the documentation for information on both the change process and hardware ordering procedures.
    Tools it asked for
    • search_docs {"question":"change process or procedure for ordering hardware"}
    • search_docs {"question":"hardware ordering procedure"}
    Why it stopped
    It asked for a tool. The loop runs the tool and calls the model again, unless the cap is reached.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 329 of the trace

  7. docs worker, call 2Answered

    Time8.429 sTokens1,575 read, 411 written
    What happened, and how it is measured
    What the model wrote
    Based on the search results, the documents describe a **change management process** related to hardware work (such as line card replacements), but they **do not contain a specific procedure for ordering hardware** (e.g., procurement, purchase orders, vendor selection, etc.). Here is what the documents **do** cover regarding the **change process** for hardware-related work: ### Change Window Classes (by Device Tier) > *"Window class is determined by device tier, not by the size of the change."* - **Tier 1 devices** (core routers and core switches): Require a **four-hour approved window**, submitted **no later than ten business days** before the change date. The window must be **approved by the change advisory board and by the on-call network lead** for the region. - **Tier 2 devices** (edge routers, aggregation switches, and route reflectors): Require a **two-hour window** submitted **five business days ahead**. - **Tier 3 devices** (access switches, out-of-band console servers, and lab equipment): Changes may be made during the **standing weekly window with no separate submission**. ### Emergency Changes > *"Emergency changes are exempt from the notice periods above. An emergency change requires verbal approval from the on-call network lead and a written record filed within twenty-four hours."* ### Hardware-Specific Notes > *"The NX-4200 does not support hot insertion on slot 0. Slot 0 requires a chassis reload, which converts this into a Tier 2 four-hour change."* --- **In summary:** The documents cover the change management process for performing hardware changes in the network, but they **do not contain a procedure for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
    Why it stopped
    It finished and gave its answer.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 330 of the trace

  8. Back to the orchestrator: an answer
    What the worker passed back
    Based on the search results, the documents describe a **change management process** related to hardware work (such as line card replacements), but they **do not contain a specific procedure for ordering hardware** (e.g., procurement, purchase orders, vendor selection, etc.). Here is what the documents **do** cover regarding the **change process** for hardware-related work: ### Change Window Classes (by Device Tier) > *"Window class is determined by device tier, not by the size of the change."* - **Tier 1 devices** (core routers and core switches): Require a **four-hour approved window**, submitted **no later than ten business days** before the change date. The window must be **approved by the change advisory board and by the on-call network lead** for the region. - **Tier 2 devices** (edge routers, aggregation switches, and route reflectors): Require a **two-hour window** submitted **five business days ahead**. - **Tier 3 devices** (access switches, out-of-band console servers, and lab equipment): Changes may be made during the **standing weekly window with no separate submission**. ### Emergency Changes > *"Emergency changes are exempt from the notice periods above. An emergency change requires verbal approval from the on-call network lead and a written record filed within twenty-four hours."* ### Hardware-Specific Notes > *"The NX-4200 does not support hot insertion on slot 0. Slot 0 requires a chassis reload, which converts this into a Tier 2 four-hour change."* --- **In summary:** The documents cover the change management process for performing hardware changes in the network, but they **do not contain a procedure for ordering or procuring hardware**. You may need to consult a separate procurement or supply chain policy for that information.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.

    Line 331 of the trace

  9. Orchestrator, call 2Answered

    Time7.354 sTokens1,470 read, 377 written
    What happened, and how it is measured
    What the model wrote
    Here's a combined summary for you: --- ## Restocking Mice | Detail | Value | |---|---| | **Current Stock** | 0 units | | **Price per Unit** | $19.50 | | **Units Needed** | 10 | | **Total Cost** | **$195.00** | You're completely out of stock, so you'd need to order all 10 units at a total cost of **$195.00**. --- ## Change Process for Hardware The network operations documents cover the **change management process** for performing hardware-related changes (not procurement/ordering specifically). Here's how it works: - **Tier 1 devices** (core routers & switches): Require a **4-hour approved window**, submitted **10 business days** in advance, approved by the **Change Advisory Board** and the **on-call network lead**. - **Tier 2 devices** (edge routers, aggregation switches, route reflectors): Require a **2-hour window**, submitted **5 business days** ahead. - **Tier 3 devices** (access switches, console servers, lab gear): Can be done during the **standing weekly window** with no separate submission. - **Emergency changes**: Exempt from notice periods but require **verbal approval** from the on-call network lead and a **written record within 24 hours**. > ⚠️ **Note:** The documents don't include a specific *procurement/ordering* procedure for hardware. You may need to consult a separate procurement or supply chain policy for that step. --- Let me know if you'd like to dig deeper into either topic!
    Why it stopped
    It finished and gave its answer.
    How it is measured
    Time: a timer started before the model call and stopped when the reply arrived. It includes the network. Tool runs and the time between calls are not timed.Tokens: the counts the model provider returned with this reply.

    Line 332 of the trace

4. What went wrong, and how to fix it

Each finding is worked out from the trace on this page.

Run A

  • A worker ran out of steps

    What happened
    The inventory worker used its 2 calls and stopped before it finished. It reported that it did not finish.
    Why it matters
    The answer for that part of the question is missing. Because the worker said so, the orchestrator knows, and can retry or say what is missing.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
    How to improve it
    Compare with a run at a higher cap on the same question: if the worker finishes there, the cap is too low for this kind of question.

    Show the step

  • A worker ran out of steps

    What happened
    The inventory worker used its 2 calls and stopped before it finished. It reported that it did not finish.
    Why it matters
    The answer for that part of the question is missing. Because the worker said so, the orchestrator knows, and can retry or say what is missing.
    How it is measured
    The handoff's status is not “ok”, and the worker's last call carries the cut-off mark.
    How to improve it
    Compare with a run at a higher cap on the same question: if the worker finishes there, the cap is too low for this kind of question.

    Show the step

  • The orchestrator asked the same worker again

    What happened
    It asked the inventory worker 2 times. The later attempts cost 2 more model calls.
    Why it matters
    A retry can rescue an answer, or repeat the same failure at extra cost and time.
    How it is measured
    Counted from the handoffs: more than one handoff from the same worker in one session.
    How to improve it
    If the retry fails the same way, retrying is not the fix. Look at why the first attempt stopped.

    Show the step

The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.

Run B

Nothing in this trace was cut off, missing or retried.

The link holds your text, the settings and the result, so anyone who opens it can run the same check again. It is made in your browser: nothing is sent. Anyone you give it to can read your text.

The whole recorded run

All 48 sessions, grouped by their settings. Counted on this page from the same file.

Worker capWhen the cap is reachedSessionsModel callsCalls cut offAnswers reported missingUnfinished answers passed on
2honest1810310100
2silent189811011
5honest633000
5silent632000
All sessions48266211011

The run's own script printed these totals when it finished: 266 model calls, 21 cut off, 83 handoffs, 10 with an empty answer. This page counts 266, 21, 83 and 10 from the same file.

These counts describe this one run: six questions, repeated three times at cap 2 and once at cap 5. They are not a rate for any model.

Not in these traces

These were not recorded, so this page shows no value for them.

  • How long the person waited

    UNMEASURED

    Only model calls were timed. Tool runs and the gaps between calls were not, so the summed model time is not the waiting time.

  • What a run cost

    UNMEASURED

    Token counts are recorded. A cost needs a price table checked against the provider's current page, and none has been checked here.

  • What the document search returned

    UNMEASURED

    The trace records the search request, not the passages that came back, so this page cannot show which passages the docs worker read.

Where this data comes from

File
runs/ch9-caps-2026-09-05.jsonl
Repository
sanovinci/dev-to-ai-engineer (private)
Commit
eb44af88398717da4a6433ba032c59c9d85519aa
Script that recorded it
scripts/ch9_run.py
sha256
b83f2757662ee768e02dfb9e8fe4bdaa7206220fe1f432c799a2dce99f50001d

Download the exact file

The file is copied byte for byte, and the build stops if its sha256 differs from the one shown here.