- Teams heavily favor observability over evaluation: 89% instrument agents, but only 52.4% run offline evals and 37.3% run online evals.
- Observability shows what an agent did (traces, tool calls, latency), while evals determine whether the output was actually good—only evals protect quality in production.
- The gap is most dangerous in production: 22.8% of live agents run no evaluation at all, despite 94% having observability.
- Observability is adopted first because it delivers immediate debugging value, whereas evals require defining success, building/maintaining test sets, and choosing judging methods.
- Clean traces and green dashboards can mask a drifting or incorrect agent; without scoring against a standard, you can’t detect quality regressions from prompt or model changes.
Most teams instrument their agents before they ever grade them: 89 percent run observability, only 52 percent run evals.
Observability tells you what the agent did. Evals tell you whether what it did was any good. Those are different questions, and only one of them protects you in production.
The gap is widest exactly where it hurts most, on agents that are already live and making decisions for real users.
LangChain ran its State of Agent Engineering survey from mid-November to early December 2025, with 1,340 responses, mostly from technology companies. The headline number that should make you uncomfortable is not about model choice or framework wars. It is the spread between two practices that sound similar and are not.
Eighty-nine percent of respondents have some form of observability. Sixty-two percent have detailed tracing, the kind that lets you inspect each step and tool call. Among teams running agents in production, observability climbs to 94 percent and full tracing to 71.5 percent. That is genuinely good. Five years ago, shipping software with that level of runtime visibility was rare. Teams learned the lesson.
Then look at evals. Just 52.4 percent run offline evaluations against a test set. Online evals, the kind that score real traffic as it happens, sit at 37.3 percent. Even among production agents, where you would expect the discipline to be tightest, 22.8 percent run no evaluation at all. Almost a quarter of agents that are live, talking to users, calling tools, spending money, are graded by nobody.
That is a 37-point gap between watching and grading. And it is not an accident of immaturity. It is a predictable consequence of how the two practices feel to adopt.
Why the easy one wins
Observability gives you something the moment you turn it on. You add the SDK, you get traces, and suddenly the black box has windows. You can see the agent reason, watch it pick a tool, follow the chain when it goes sideways. When something breaks at 2am, the trace is right there. The payoff is immediate and obvious, and the LangChain report names exactly this: without visibility into how an agent reasons and acts, teams cannot debug failures or build trust. So they reach for visibility first. Sensible.
Evals give you nothing for free. Before you get a single score, you have to decide what good output even means for your task. You have to build a test set. You have to choose a method, and the survey shows teams splitting across human review at 59.8 percent and LLM-as-judge at 53.3 percent, often both. You have to keep the test set current as the product changes. None of that produces a dopamine hit on day one. It produces a number, later, that you then have to act on.
So one practice feels like installing a dashboard and the other feels like writing a curriculum. Of course the dashboard wins the first sprint. The problem is that teams stop there and call it done.
What observability cannot tell you
A trace is a recording of what happened. It is not a verdict on whether it should have happened.
Your agent can produce a flawless trace of a completely wrong answer. Every step logged, every tool call clean, every latency green, and the final output confidently incorrect. Observability shows you a healthy-looking run. It has no opinion about correctness, because correctness is not something you can read off a span. You only know the answer was wrong if something compared it against what right looks like. That something is an eval.
This is the trap. A good observability setup makes a broken agent feel safe. The graphs are green, the traces are tidy, errors are caught and retried. The system looks like it is working. Meanwhile the actual quality of the output, the thing the user receives, drifts with every prompt tweak and model update, and nobody has a number that moves when it gets worse.
I have written before that the human's two jobs around an agent are to set the bar and to hold the bar. Set the bar means specifying what good looks like, up front. Hold the bar means verifying, after, that the work cleared it. Evals are how you hold the bar at scale. Observability is how you see the work. Watching work go by is not the same as holding it to a standard. A team with 94 percent observability and no evals has built an excellent window into a room where nobody is checking the output.
The fix is less heroic than it sounds
You do not need a perfect eval harness to close most of this gap. You need to start grading.
Begin with the failures you already have. Your observability setup has been quietly collecting the runs that went wrong, the ones you fixed by hand, the outputs a user complained about. That is a test set waiting to be named. Pull twenty of them. Write down, for each, what the right answer would have been. You now have an offline eval that costs you an afternoon and will catch the next regression before a user does.
Pick one method and accept that it is imperfect. Human review is the most trusted and the least scalable. LLM-as-judge scales and drifts, so you check the judge against human ratings every so often. Most teams in the survey run both, and that is the right instinct: judge for coverage, humans for the cases that matter. The mistake is waiting until you can build the sophisticated version. A rough eval that runs on every deploy beats a perfect one that never ships.
Then move at least one eval online. Offline evals catch regressions before release. Online evals catch the failures your test set never imagined, the ones real traffic invents. Only 37.3 percent of teams do this, which means it is still a place you can get ahead. Score a sample of live runs, even a small one, and watch the number over time. When it drops, you find out from your own dashboard instead of from a churned account.
The teams that win the next year of agent work are not the ones with the most traces. Tracing is table stakes now; 89 percent have it. The edge belongs to the teams that turned all that visibility into judgment, that built the loop where seeing a problem leads to scoring it leads to fixing it. Observability without evals is surveillance without standards. You are watching very carefully while quality decides for itself where to go.
Sources: LangChain, State of Agent Engineering; LangChain blog, Agent Observability Powers Agent Evaluation.
If you’re running an agent, treat observability as necessary but not sufficient: add evals that explicitly define “good” for your task and measure it regularly. Start with a small, representative test set for offline evals, then layer in online scoring for real traffic where possible, using human review and/or LLM-as-judge depending on risk. Most importantly, wire the scores into your release process so prompt tweaks, model swaps, and tool changes can’t silently degrade user-facing quality.