Where the tokens go in an agent session

I spent a while building tooling for coding agents while assuming the thing I should optimise was output size. Return fewer lines. Trim the file listing. Compress the search result. It seemed obvious. The tool prints text into the conversation, the conversation costs money, so smaller output costs less.

Then I actually decomposed a session, and the picture was not the one I had in my head.

What a turn actually bills for

A chat with a model is stateless underneath. The model does not remember the previous turn. Every time the agent takes an action, the entire conversation so far is sent again: the system prompt, the tool definitions, every file that was read, every command that was run, every result that came back.

Providers soften this with prompt caching. The unchanged prefix of the conversation gets cached, and re-sending it is much cheaper than sending it fresh. Cheaper is not free. You are still billed for every cached token, on every single turn, for the rest of the session.

So a turn costs roughly the size of everything resident in the conversation at that moment. Not the size of what just happened. The size of everything that has happened.

Which gives you a rough model of a whole session:

session cost  ≈  number of turns  ×  average resident context

Two terms. Worth asking which one you can actually move.

The resident context only grows

Look at what sits in that second term.

• The harness system prompt. Fixed, and you do not control it.
• Your project instruction files, loaded at the start.
• The schema for every tool the agent has available.
• Every file read so far, in full.
• Every command and its output, in full.

Only the last two grow during the session, and they grow in one direction. Nothing leaves the conversation. Reading a file is not a one-time cost of that file. It is a subscription: you pay for those lines again on every remaining turn.

The tool schemas deserve a specific mention, because this one surprised me. A tool definition is not billed once when you install it. It is part of the prompt, so it is billed on every turn like everything else. Connect a server that exposes a dozen richly documented operations and you have added rent to the whole session, paid whether the agent calls any of them or not. A tool surface is not a menu you browse for free. It is a standing charge.

The practical consequence is that the resident term is mostly not yours to shrink. You can be careful about what gets read. You can keep your tool surface small. Beyond that, it is a floor that rises.

Output is a rounding error

Here is the part that killed my original assumption.

What the model writes is a small fraction of what a session bills. Not a modest fraction. Small enough that halving it changes almost nothing. The reason is structural: output is generated once, then it becomes part of the conversation and is re-billed as cached context on every subsequent turn, at a much lower rate. The single largest share of a long session is re-reading context that was already established.

So the effort I put into trimming a tool's output was aimed at the term that mattered least. A tool that prints twenty extra lines has added twenty lines to the resident context. Real, but marginal.

A tool that causes one extra turn has re-billed the entire conversation.

Turns are the lever

If the resident context mostly cannot shrink, and output barely counts, then the number of turns is what you have left. And unlike the other two, it responds to design.

This reframes what a good tool for an agent looks like. The question is not "how much did it print." The question is "did the agent have to ask again."

A search that returns forty file paths with no way to tell which matters causes the agent to open several of them to find out. Each open is a turn, and each opened file joins the resident context permanently. The search looked cheap. It was not. It was the most expensive thing in the sequence, because of what it made happen next.

A search that returns eight paths, says which ones define the thing you asked about rather than merely mentioning it, and shows enough of the relevant lines to judge without opening the file, can end the question in one turn. It printed more. It cost less.

Same logic in the other direction. Ten small precise calls are worse than two calls that each carry more. Batch what you can. Answer the follow-up before it is asked.

Measuring your own

None of this needs to be taken on faith. If you use a harness that writes session transcripts to disk, the numbers are already there.

Each assistant message carries a usage record with separate counts for fresh input, cache reads, cache writes, and output. The total billed for that turn is the sum of all four. Two things to be careful about. Dedupe by message id first, because a streamed message can appear more than once and double counting will flatter or wreck your result. And group by turn, so you can watch the resident context climb rather than seeing one aggregate.

Then plot the per-turn total across the session. You are looking for two things: how fast the line rises, which tells you what is accumulating, and how many turns there are, which is the thing you can act on. Compare the same task done two ways and count turns, not tokens. Turns are the honest metric because tokens follow from them.

I would suggest doing this on your own sessions rather than trusting anyone's published figures, mine included. The shape holds across harnesses. The exact proportions depend on your system prompt, your instruction files, how many tools you have connected, and how large the files in your repository are. A codebase with long files behaves differently from one with short ones.

What I changed

The measurement changed what I build. I stopped trying to make output smaller and started trying to make answers final.

In practice that meant a few things. Ranked results instead of a list, so the agent does not open five files to find the one. Enough of the matching lines inline that the file often does not need to be opened at all. Relationships between files returned alongside the file, because "who calls this" being a second question is a second turn. And a deliberately small tool surface, since every operation I expose is charged on every turn of every session whether it earns that or not.

That last one is why coldstart is two commands rather than the larger set I started with. find locates the files for a concept and ranks them by evidence. gs takes one file and returns its shape along with who uses it. There is no third operation, and cutting the others was not a simplification for its own sake. Each one was rent.

The general form of the lesson is short. Work out the cost model of the thing you are building for before you optimise anything, because the obvious target and the real one are often not the same, and in this case they are not even close.