Building Production Agents That Actually Ship

Most agent demos never ship. The gap between a working notebook and a system that survives a Monday morning is wider than teams expect — and it's rarely about the model.

#The four failure modes

After deploying agents for a dozen clients, we see the same four categories of production incidents:

1. Cost blowouts — a runaway loop burns $400 in an afternoon.
2. Silent regressions — a model update quietly changes tone or accuracy.
3. Tool timeouts — a 30s API call cascades into user-visible failures.
4. Prompt drift — someone edits a prompt without evals and the whole thing gets worse.

#A minimum viable production stack

Here's the smallest set of pieces that has worked reliably for us:

from langgraph.graph import StateGraph
from langgraph.checkpoint.postgres import PostgresSaver

# 1. Persistent state — recover from crashes
checkpointer = PostgresSaver.from_conn_string(DB_URL)

# 2. Budget guardrails per run
class AgentState(TypedDict):
    messages: list
    tokens_used: int
    max_tokens: int  # hard cap

def budget_gate(state):
    if state["tokens_used"] > state["max_tokens"]:
        raise BudgetExceeded()
    return state

Wire that into every node and you eliminate 80% of runtime surprises.

#Evals are the product

The temptation is to treat evals as a QA activity. In practice, your eval suite is your product spec. If you can't articulate a scenario in an eval, you can't guarantee it in production.

We run three layers:

Layer	Cadence	Purpose
Unit evals	Every PR	Individual tool + prompt behaviour
Regression evals	Nightly	Full task suite against golden traces
Shadow evals	Continuous	Live traffic replay on candidate models

#What we'd tell our past selves

Ship the ugly version behind a feature flag, wire up traces before your third prompt tweak, and never let anyone change a prompt without an eval delta in the PR. Everything else is decoration.