Your agent solved this yesterday. Today it paid full price to solve it again.

I have been building Basic2202 for a few months. The hard part was not the part I expected. This post is about both: the problem, and the measurement problem hiding underneath it.

The problem

An agent is a program you give a task to in plain words. It works out a plan, calls some tools — looks things up, reads files, asks other systems questions — reads what comes back, and answers.

Here is a task I run often:

check the eu-west-1 fleet and tell me which device is most at risk

Two tool calls. List the devices in that fleet, then get the details of the riskiest one. Then answer. It works, it costs what it costs, fine.

Next week I ask the same thing about us-east-2.

The agent works the plan out again. From nothing. It has no memory of having solved this, and nothing in the stack is arranged to use that memory if it had. Every token it spends rediscovering list the fleet, then look at the worst device is a token you already paid for. Last week.

Two things travel together in every run: the plan and the values.

• The plan: list the fleet, then look at the worst device.
• The values: eu-west-1, last Tuesday, customer 4471.

The values change all the time. The plan barely changes. And you pay full price for both, every time.

In my own account the task was recorded as check the us-east-2 fleet, and the agent did it with one more look-up: three tool calls, listing the fleet and then looking up both of its devices, not only the riskiest. Same shape. One row per task, and the column that matters is Similar hits: this task came back five times.

This is not true of all agent work, and I will get to where it is not, because that part matters. But it is true of one very common shape: scheduled sweeps, recurring reports, per-customer checks that always start the same way, and anything you run across forty repos or regions or accounts.

Record, calculate, replay

Three steps. You can stop and look at the output of each one before you trust the next.

Record. A small program called bir sits between the agent and its tools and writes down what happens: the words you typed, every tool call and its arguments, what came back, how long it took, what it cost, and the final answer. It runs the calls the model was going to make anyway and hands back the results unchanged. The model does not behave differently; the one trace of the runner it can see is a reserved property, __bir_call_id__ ("Reserved. Leave unset."), declared on each wrapped tool's schema.

One honest detail. bir speaks MCP, the protocol most agent tools use, so it sees those calls natively. An agent's built-in tools, like reading a file or running a shell command, are captured by its second half, a recorder behind Claude Code's hooks, which starts itself with each session once the project is set up. A recording made without it is labelled partial, because a partial record that looks complete is worse than none.

What the recorder sends is what the agent did, with the secrets it recognises stripped out first: values under names like token, password or api_key, everything inside an env block, and strings shaped like API keys, bearer tokens, private keys, passwords in URLs, cookies or .env lines, in the request you typed and the answer as well as in the tool calls. It travels over HTTPS to your own account, where you can read it, and so can an administrator of the service, read-only; you can delete it at any time. The full statement, including what runs without asking you, is the handbook's Security and privacy section.

Calculate. A recording is a transcript: what happened, once, with those exact values in it. From it, Basic2202 works out a different thing, which it calls a scenario. A scenario has three parts:

• an intent: what this task is, in one sentence;
• the parameters: which values were specific to that run (us-east-2) and what they mean;
• one rule per step: how to work out that step's arguments from new values, and how to read what comes back.

The line I keep coming back to: the transcript says what the agent did; the scenario says what the agent was doing. Only the second one is worth running again.

This step can go subtly wrong, so there is a free dry run that checks the derived rules against the recorded outputs without calling anything. The mistake to look for is a step that remembers what it should work out: a step that sends dev_4411 because that is what the recorded run looked up, rather than because it read the device list the first step just returned. Identical on the recorded run. Wrong on every other.

In this scenario, step 0 lists the fleet, and its input rule sends whichever fleet the new request names. The rule for step 1 is one line:

return { id: (respParams.orderedDeviceIds || [])[0] ?? parameters.id };

It sends the first device id from the list step 0 just returned, live. Step 2 does the same with the second. Only if that list is too short does it fall back to the id parameter. Nothing is remembered.

Read a dry run with that in mind. Asked about eu-west-1, it sends eu-west-1 to step 0, but steps 1 and 2 still show dev_4411 and dev_7b22. That is not the mistake: nothing was really listed, so they read the recorded list. In a dry run, check the rule, which says where the id comes from, not the ids it prints.

One honest limit. The task was recorded on a fleet with two devices, so the scenario has exactly two look-ups. Reading the rules, on a fleet of three it would look up the first two devices and never the third. That is the kind of thing to read for before you trust a scenario.

Replay. When a new request means the same thing as a recorded one, the plan runs directly. The match is made on the wording of the request, not on the transcript and not on the answer, so it is settled before any work starts. Where every step is a tool bir can call itself, the whole plan runs with zero model tokens, and the model's only job is to read the results. Where some steps are not, the live session is steered instead: each step runs as normal, with its arguments pinned to what the scenario worked out.

One replay of the fleet task cost $0.157 against the agent's $0.249, and saved $0.092.

This is not a cache, and not a semantic cache. The semantic caches in AI gateways such as TrueFoundry's and Gravitee's return a stored reply when a new prompt looks enough like an old one; the model is never called, and the reply is as old as the cache entry. Here the tools are really called, against live data, and the answer is about today. What is cached is the plan, not the answer, which is why I call it scenario caching. What gets skipped is the working-out, not the work. The handbook has the two side by side: Scenario caching, not semantic caching.

The part that turned out to be hard

How do you know any of that saved anything?

I assumed this was the easy half. It is not, and I now think it is the question most things in this space are quietly not answering.

A saving is a subtraction: baseline minus cost. What the task costs when the agent does it the ordinary way, minus what the replay actually spent.

So what happens when you do not have a trustworthy number on the cost side?

In practice an unmeasured replay still has something priced, usually one small call. So the subtraction still runs. And it books nearly the whole baseline as profit.

It does not error. It does not look wrong. It produces a large, confident, fictional number, and it produces one for every unmeasured run. Add those up and you get a headline that is very impressive and worth nothing.

So the two are kept apart, permanently:

The headline is deliberately the smaller number. The gap between the two is not a defect being worked on. It is the point.

Two more rules follow the same instinct:

A baseline drawn from one run is labelled as one run. Every saving on a row is measured against what that task costs the agent. If that figure comes from a single sample it is a guess, not a statistic, and the console says so.

A scenario is allowed to lose money, out loud. If a scenario's recent replays cost more than the agent does, the row is flagged as not paying off. Savings are not clamped at zero. A negative is a real outcome, and it is the one you most need to see. A scenario quietly costing you money while reporting a cheerful zero is the worst thing the system could do to you.

And the bit that makes this checkable rather than merely stated: every saving sits on the row of the run it came from, in the recordings list, next to the baseline it was measured against. The verified figure is deliberately the smaller one. You can read along any row and work it out yourself: what the task costs the agent, what a replay costs, and how many replays there were. That is the only way a savings claim is worth anything.

Which is also why this post has no headline savings number. The screenshots are from my own demo account, and the figures on them are tiny: cents, on a three-step task. I have bigger numbers. Showing them would be the normal move, and I think it is the least trustworthy move available to me, because you have no way to check whether I split verified from projected before I put a number on a slide. The mechanism is the claim. Measure it on your own work.

Where this does not pay off

Worth being direct about, because if your work is the wrong shape you will find out in three weeks anyway and be annoyed.

Exploratory work gets nothing. Debugging, code review, research: anything where the second tool call depends on what the first one said in a way you could not have predicted. There is no plan to extract, because the plan is the work. "Figure out why the build broke" is a different investigation every time, even when the words are identical.

Anything that must not run unattended. A replayed scenario is a pre-approved list of tool calls with computed arguments. A steered step is answered "allow", so a pinned shell command runs without the permission prompt you would otherwise see. If that sentence makes you uneasy about a particular tool, that instinct is correct: keep it off the list.

Short tasks. Under four tool actions there is not much plan to capture. Those runs are hidden unless the same request keeps coming back, and they were already as cheap as they get.

Work you run twice a year. The environment has moved on by the time it comes round, the scenario no longer fits it, and rebuilding it costs more than the run saves.

If none of your work has the right shape, Basic2202 will record it faithfully and save you nothing, and I would rather you knew that now.

Recording is useful on its own

The thing I did not expect: for some teams the recorder is the whole value, and replay is a question for later.

A team that never calculates a single scenario still ends up with a priced, searchable record of what its agents actually did: every run, every tool call, what it cost, how long it took. If you currently have a line on a bill that says "agents did work", that record is available to you without ever replaying anything; bir replay off keeps the runner to recording.

Trying it

It is free. Recording, calculating, replaying, as many scenarios as you like, and none of them expires. There is no card and no limit to run into, so the only question left is whether it pays off on your work rather than on my benchmark.

What I sell is the other deployment. If your recordings cannot leave your company — and a recording holds every tool call and everything that came back, so for a lot of companies they cannot — there is Basic2202 In-house: the same service in your own AWS account, for a fifth of what it saves you. You keep four fifths, only the verified saving counts, and a month that saves nothing costs nothing. It is not something you can install yet, so if that is you, talk to us at support@bi2202.com. Your recordings never leave your AWS account: scenario calculation calls Claude through Amazon Bedrock inside it, and the only thing that leaves the server is one number an hour, the dollars it saved you, which goes to AWS for the bill.

npm install -g @basein/runner
cd the-project-you-start-claude-code-in
bir setup --auth-url https://api.bi2202.com   # signs in, wires the project, starts the recorder
claude

Or paste the one line from the console's Set up the runner page, which does the same. Claude Code asks about the project's MCP server the first time; say Yes, then run bir setup again so the runner can route it.

If you do not have a project with an MCP server, the handbook's step 2 hands you a real one: two files, about a minute, no dependencies. The handbook also shows every one of these screens, so you know what "working" looks like before you start.

Then pick the most boring repeated task you have, the weekly sweep nobody enjoys. Record it a few times, let it derive the plan, dry-run it, and read the ledger. If the verified number is small, that is the true answer, and it cost you an afternoon.