Forward-Deployed Reliability · Case Study
Anyone can build a classifier. The FDE job is proving what happens on the day it fails silently — before the regulator finds out first. This is that project, built for one imaginary bank and eight deliberate ways to break it.
A reliability harness for an LLM complaint-triage agent. The classifier is deliberately simple — the value is the machinery around it that detects when it’s silently wrong.
Two things. The agent emits {team, urgency, confidence, ambiguous, rationale} per complaint. The project outputs how the harness does against 8 planted failures — what it catches, what didn’t fire, and where I got it wrong.
The forward-deployed judgment that matters: profile data before building, predict failures before coding, measure detection not accuracy, and report honestly — on real regulated-domain data.
Not “95% accurate.” Anyone can screenshot an accuracy number. Almost nobody can show you the failure that keeps reporting 95% while the system is quietly broken in one specific place. That failure is the whole project.
01 · The customer
A mid-size bank. Forty branches. A six-person complaint-ops team that reads every consumer complaint by hand and decides three things: what product it concerns, how urgent it is, and which internal team owns it. Volume is a few hundred a week and climbing.
The stakeholder is Dana — eleven years at the bank, six running this team, not technical, owns the regulatory exposure. She has sat through a vendor demo that went well in the room and badly in production. Her posture isn’t hostility. It’s scar tissue.
“What happens when it’s wrong, and will I find out before the regulator does?” — Dana’s actual question. Not “is the AI accurate.”
Accuracy is the fourth thing on her list. That reordering is the whole project.
02 · Success in her terms
A mediocre agent with rigorous failure analysis proves more than an excellent agent with none.
03 · The core artifact
For each mode: what it is, how I inject it, and what detection should look like. The verdict tags are my prediction, recorded before running the harness. The two I expect to miss are the interesting half.
The agent calls a lookup tool with an argument shape the tool no longer accepts. Inject: rename a field without updating the prompt. Should surface as a hard error at the gate — never silent.
The agent retrieves the wrong taxonomy slice and answers anyway. The tell is the shape: accuracy collapse concentrated in the affected categories, not spread evenly.
System prompt says one thing, few-shot examples demonstrate another. The agent silently follows the examples. Systematic deviation on one rule while overall accuracy stays high.
Emits an owning team or urgency tier outside the defined set. Should surface as schema-validation rejection plus a count of how often it’s attempted.
A long complaint is cut before the part that matters — and legal threats appear late. Urgency accuracy drops on long inputs while product accuracy holds. Invisible unless the eval stratifies by length.
Same prompt, new model version, shifted output distribution. Should fire a distribution-shift alarm on a frozen baseline. Expect false positives on small samples — measured and reported honestly.
A complaint spans two categories; the agent picks one with high confidence and the other team never sees it. A calibration gap on cases even the labeler couldn’t decide.
The model declines a distressing complaint. In a triage system a refusal is a dropped complaint — worse than a wrong answer. Refusals must count as failures and land in a human queue, never disappear from the metric.
04 · What the data actually looked like
Working set: 13,991 CFPB complaints with narratives, salvaged from a truncated API stream. Three findings, each of which changed the design before the agent was written.
The CCDB endpoint accepts size and frm and silently ignores both — every bounded request streams until the connection dies mid-record. So ingest cuts the stream deliberately, salvages complete records, and counts the fragments it drops. salvage.py reports dropped_fragments=1 on every run — a number that should never silently become 40. Build ingest that degrades honestly instead of failing whole.
| Count | Product |
|---|---|
| 10,285 | Credit reporting / consumer reports |
| 1,054 | Debt collection |
| 866 | Credit card or prepaid card |
| 713 | Checking or savings |
| 404 | Mortgage |
| 224 | Vehicle loan or lease |
| 109 | Student loan |
A random 80-case sample would hold ~59 credit-reporting complaints and one student loan. The golden set must be stratified by product — over-sample the rare ones, report per-class accuracy, never an aggregate.
Failure mode 5 was a hypothesis written before looking at data. Then measured: of 1,419 legal-threat complaints, 778 — 55% — carry that signal in the final 40% of the narrative. Under naive truncation, more than half lose their most important attribute while the classifier keeps working and reports high confidence, because the product category is set in the first paragraph.
11,214 complaints (80.2%) contain XXXX — the CFPB’s inline PII scrubbing. “I paid $XXXX on XX/XX/XXXX to XXXX and they never credited it” has had every specific removed. So: no amount extraction, no date extraction — building them produces confident nonsense. Product, urgency, owning team, confidence, and rationale all survive redaction. On a real deployment this flips: Dana’s text has more PII, so the problem becomes “scrub before it reaches the model.”
05 · The result
80-case stratified set, real inference on gpt-5.6-sol, plus a 30-case LLM-assisted gold pass. A council attacked this on eval rigor and label circularity. Two findings got worse for me and better for you as a result. Of the 5 failure modes that actually manifested, all 5 were detected — including one the aggregate metric missed.
People tell the story first, then threaten to sue at the very end. Cut the complaint short and the model never sees the threat.
▶ Try the live demo — pick a real complaint, inject the truncation failure, watch the harness catch it.
Cap input at 600 chars on 31 legal cases. Urgency drops to 19.4% (below the baseline 95% CI of [61,90]) while team holds. But the control re-scores the agent against the truncated text and it jumps to 67.7%: the model is mostly right about what it saw. This is upstream information loss, not model error, and no model upgrade fixes a starved input.
Drop the DEPOSITS definition and its accuracy goes 58% → 0% while every other class holds. Aggregate team accuracy barely moved, so my first detector called it a miss. That is the exact trap this project preaches against, sprung on me. I added per-class collapse detection and it catches it.
A prompt addendum demotes attorney/CFPB mentions to tier 1. Urgency drops to 0% while team holds. Caught because urgency is scored separately on the attacked slice; on an aggregate metric it hides.
Rename every team code. 23 of 24 outputs fail validation and stop at the gate.
On cases the labeler couldn’t decide, the agent is more confident (0.97) than on clear ones, and self-flags only 36% of them. The calibration gap, from existing data.
Offered only 3 of 8 teams, the model used the valid UNROUTABLE escape hatch and invented zero codes. Safe degradation to a human queue, not silent error. A guardrail worked, so I report that instead of staging a catch.
15 distressing complaints (abuse, death, threats) routed through. 0 refusals. The failure didn’t manifest on this model.
The backend authenticates one model through a personal CLI. Drift needs a second model, so this one honestly cannot be exercised here. Naming the gap beats faking it.
Null control fires on nothing (0 false positives). Full write-up in docs/03-results.md.
06 · So what — the business case
A technical finding isn’t a business case. Here’s the translation, in the three buckets that actually get budget: risk mitigation, cost savings, and the enabling value.
Worked example, stated assumptions — a mid-size bank taking ~500 complaints/week (~26,000/year). These are my numbers, not a client’s. The point is the method.
Every legal-escalation complaint is a 48-hour clock in a regulated process. The truncation failure doesn’t make the system look broken — product classification stays at 71% and confidence never moves — it just quietly stops flagging urgency. At this volume that’s roughly 1,480 missed escalations a year, each one arriving at the regulator instead of the ops queue. The harness is what surfaces that before it becomes an enforcement conversation.
A six-person team reading every complaint by hand. At ~5 minutes each, 26,000 complaints is ~2,170 hours a year. The agent routes confidently where it can and abstains where it can’t — and the abstain rate is the dial. At 70% auto-routing, that’s roughly 1,500 hours returned to the team, with the remainder landing in a human queue by design rather than by accident.
| Artifact | What it’s for |
|---|---|
| Golden set | an agreed definition of “correct,” stratified so rare categories aren’t invisible |
| Eval gate | wired into CI — a metric below its baseline CI blocks the deploy |
| Audit row | input hash, model + prompt version, output, confidence, one-line rationale — owned by compliance |
| Abstain path | anything unroutable lands in a human queue; the pressure-release valve that makes it safe to turn on |
| Drift alarm | you hear it from the system, not from an auditor |
The honest limit: I built the detection layer first because you can’t price a risk you can’t see. Turning these into real numbers needs the client’s actual volume, their loaded cost per triage hour, and their own history of what a missed escalation costs. That’s a conversation, not an assumption — but it’s the conversation that gets the thing funded.
07 · If this were a real Monday
This is offline eval on a public dataset, on purpose. But an interviewer will ask what the real deployment looks like, so here it is, plainly.
UNROUTABLE path is not a failure, it is the pressure-release valve that makes the whole thing safe to turn on.None of this is built here. Naming it precisely is the point: an FDE who can’t sketch the integration path in a minute has built an offline eval and nothing else.
08 · Miscalibrated confidence is the failure this project exists to catch
Stated up front, not as a disclaimer:
A mediocre agent with rigorous failure analysis proves more than an excellent agent with none. — the thesis of the whole project