Should I use an agent framework?

For small tasks a framework is usually too much: a loop, a tool list and a state object will do. Frameworks earn their place on long, multi-step, parallel flows. The thing to judge them on is how easy it is to pause a run, inspect it, and re-execute a single step.

How do I measure whether an agent works?

Two separate measurements: was the final output correct, and was the sequence of steps sensible. Without the second, an agent that reaches the right answer by the wrong route looks correct — and collapses the moment the input shifts.

How much testing before production?

Fifty scenarios drawn from real usage, failure cases included. Beyond that, require human approval for an initial period and relax it gradually, in proportion to how high the approval rate stays.

← All notes

Agents

Why agent projects fail

An AI agent works a task step by step rather than answering in one shot: it calls tools and corrects itself along the way. Getting one demo-ready takes a week. What separates the ones that survive production is not the model but how the tools are designed and what happens when something fails.

Published: 2026-09-06
Built by: Zero-Shot Team
4 min read

Tools written for humans, not for the model

An agent is only as good as the tools it holds. On most projects those tools are thin wrappers over existing APIs: fourteen parameters, half of them required, and a status code for an error message.

A good tool is narrow and does one thing. It takes few parameters, its name says what it does, and its errors explain in words what to do next. 'Start date cannot be after end date' lets an agent correct itself; 'invalid request' does not.

In practice the change that helps most is usually removing tools. An agent choosing among thirty tools picks wrong noticeably more often than one choosing among eight.

Failure was never designed

In a demo every call succeeds. In production the API times out, the record is missing, the permission is denied. If the agent was never told what those mean, it either loops or ignores the failure and starts inventing.

Three things are needed: an enumerated set of errors each tool can return, a limit on how many times a step may be retried, and a ceiling on total steps. An agent with no ceiling can repeat one call all night and leave a bill in the morning.

There is no path back to a human

Agents earn the most where they do ninety percent of a task and leave the rest to a person. Most systems instead try to do all of it, or none.

Handoff points have to be written in advance: stop and ask when an amount crosses a threshold, when confidence is low, when the same step fails twice. These are not bolted on later; they are part of the flow's design.

Handoff is also an acceptance mechanism. Every step a person approves becomes a training example and an evaluation case for the next version.

Cost explodes in production

One agent run is not one model call but fifteen, twenty, sometimes fifty. The difference is invisible in a demo running ten times a day. At ten thousand runs a day the same flow can become the most expensive line item in the company.

The numbers to track from day one are total tokens and total time per run. Most steps can drop to a smaller model; only the step that actually decides needs to stay on the large one.

When not to build an agent

If the work follows a fixed sequence, an agent is unnecessary. For a five-step flow with little branching, ordinary code is cheaper and far more predictable.

Agents exist for work whose order cannot be known in advance — where which tool to call depends on what comes back. Without that uncertainty, agent architecture adds only cost and surface area for failure.

In short

Keep tools narrow and few; make error messages say what to do next.
Cap retries and total steps.
Design handoff points up front: thresholds, low confidence, repeated failure.
Measure tokens and time per run from day one; move non-deciding steps to a smaller model.
Don't build an agent for a fixed sequence — plain code is cheaper and more predictable.

Common questions

Should I use an agent framework?: For small tasks a framework is usually too much: a loop, a tool list and a state object will do. Frameworks earn their place on long, multi-step, parallel flows. The thing to judge them on is how easy it is to pause a run, inspect it, and re-execute a single step.
How do I measure whether an agent works?: Two separate measurements: was the final output correct, and was the sequence of steps sensible. Without the second, an agent that reaches the right answer by the wrong route looks correct — and collapses the moment the input shifts.
How much testing before production?: Fifty scenarios drawn from real usage, failure cases included. Beyond that, require human approval for an initial period and relax it gradually, in proportion to how high the approval rate stays.

More notes

All notes →