Know when your agent gets worse, and why.
Plumbline watches your AI agents in production. When quality slips, it tells your team in Slack or email and names the likely cause: the prompt version, the failing step, the deploy. When an agent should not act alone, it hands the moment to a person and gives the agent their decision.
Source available on GitHub. Runs on your own Postgres. Set up in an afternoon.
Two minutes, on the real dashboard.
AI agents rarely crash. They get quietly worse.
A prompt tweak, a model update from your provider or a new tool changes behaviour, and nothing errors. The agent still says it is done. Customers notice before your dashboards do, and then someone spends a day reading traces to work out which change did it.
Plumbline checks whether the job was really done, learns what normal looks like for each agent, and speaks up when that changes.
The failures that never throw an error.
The change that broke it
Every alert says when the drop started, which prompt, model or code version the bad runs share, which step began failing and which deploy went out around then. Each part has to pass a significance test, or it is left out.
"Done" that wasn't
Agents report success while the booking never happened. Plumbline compares what the agent claimed with an independent check: yours, or a judge model grading a sample of runs against your rubric.
Drift below your alert lines
Once a day it looks at what is odd but not yet alarming, such as a new version that fails more or a step that started failing, and posts a short note. Every number in it comes from your data.
Regressions before they ship
Run your eval suite in CI and block a prompt or model change that makes things worse, with the failing cases listed in the pull request.
A colleague who reads every dashboard.
Founders and product leads get the note in Slack without opening anything. If you connect a model, it connects the findings and says what to check first. It can only cite what Plumbline found, and a point with a number that is not in the data is dropped.
When the agent should not act alone, a person decides.
Before a refund, an email to a customer or anything that cannot be undone, the agent asks Plumbline. Your policies decide: low confidence, an amount over a limit, a guardrail block, a suspected prompt injection. The right people hear at once in Slack, email or your own tool. One takes it, decides, and the agent, which can wait for the answer, gets it back.
- Service levels: past the deadline the same people are told again
- Time to a person, missed deadlines and whether escalating was right, all measured
- One call from any framework, the SDKs, or MCP
Answer for your agents without assembling the evidence by hand.
The incident report, already written
Every alert that turns out to be an incident gets an account from the records: a timeline, how long it took to notice and to fix, how many runs went wrong beyond normal and what they cost, the likely cause, and follow-ups. Drafted while it is open, posted when it closes.
Proof of what your agents did
One export with every action, guardrail check and decision, every escalation and who resolved it, and every change people made. Each record is chained to the one before, and the file is signed, so a customer's security review or an auditor can check that nothing was edited or removed.
How you compare
Teams that opt in contribute to an anonymised quarterly report on how agents in production are doing. Every figure needs at least five organisations behind it, and no single one can be singled out. Your dashboard shows where you stand.
A tool your agents and assistants can use
Plumbline is an MCP server. An agent asks for a person and waits for the decision; the assistant you work in, such as Claude or Cursor, can explain why an alert fired, read the post-mortem, and pull the runs behind it.
# Claude Code. Cursor and other MCP clients take the same URL and key.
claude mcp add --transport http plumbline https://your-plumbline/mcp \
--header "Authorization: Bearer $PLUMBLINE_KEY"
Three steps, most of them automatic.
-
Connect your agent
Point your framework's tracing at Plumbline, add a few lines with the TypeScript or Python SDK, or connect over MCP. The dashboard's Connect page shows exactly what to paste and ticks itself off as data arrives.
-
It learns normal
Plumbline computes task success, silent failures, false completions, latency, cost and more than 50 other measures per agent, each day, and learns each one's usual range.
-
It tells you
When a measure moves the wrong way, your team hears about it once, with the likely cause and the runs behind it. Hard failures are caught within minutes.
# Python. Any OpenTelemetry exporter works too.
plumbline = Plumbline(URL, KEY, version={"prompt": "v12"})
with plumbline.run("stylist-matcher", request_id) as run:
shortlist = match(request)
run.end("succeeded", claimed_done=True)
Phone agents get worse in their own ways.
Plumbline watches calls from Vapi, Retell, Pipecat, LiveKit or your own stack, and tracks what only calls have.
- A quality score out of 100 for every call, with the moments that cost points marked in the transcript
- How long callers wait for each reply, and calls with dead air over 3 seconds
- Callers who hang up in the first few seconds
- Transfers to a person, frustrated callers, and callers who had to repeat themselves
- Cost per minute
- The agent version behind a change, picked up automatically from Retell
The stack you already have.
- LangGraph
- LangChain
- OpenAI Agents SDK
- CrewAI
- LlamaIndex
- Vercel AI SDK
- OpenTelemetry
- MCP (Claude, Cursor)
- Vapi
- Retell
- Pipecat
- LiveKit Agents
- Slack
- Webhooks
Founders, Leaders & Companies shipping agents to real customers.
- Teams without a dedicated evaluation or ML platform group
- Your data stays in your own Postgres; prompts and outputs are not stored unless you choose to grade them
- Alerts that fire together arrive as one message, not a dozen pings
- Set up in the browser: no config files to learn before you see value
"The failures that hurt us never threw an error."
I'm a technical cofounder and CTO for a FashionTech Startup. While building our own multi-agent system, I kept seeing agents fail silently. A run would say "done" when the agent hadn't actually done the thing. Quality would drift over a few weeks with nobody noticing. An agent would be about to take a risky action with no one checking. Logs never caught any of it.
So I built Plumbline for us: something that checks whether the job was really done, says what changed when quality slips, and puts a person in front of the decisions an agent shouldn't make alone. Once I saw how much it caught, I decided to make it available to every team shipping agents.
Try Plumbline free on one of your agents.
We're inviting a small group of engineers and engineering leaders with an AI agent in production, or about to be, to use Plumbline free and tell us honestly what's missing.
- Free for the whole beta, 60 to 90 days
- A 30-minute setup call with Shama, then a few weeks on one real agent
- A direct line to the founder, and a real say in what gets built next
- Runs in your own cloud: your data stays yours