AI agent evaluation: what an agent's stated confidence is worth

By the founder of Cerevisor. Checked on 7 October 2026 against the source code of Cerevisor 3.7.1.

On 1 October I switched off a feature I had been quietly proud of.

Some context first. Cerevisor is a desktop app where AI agents sit on a canvas as a workflow, each one handing its work to the next. An agent is an AI model given a task, some tools, and room to take several steps on its own. A harness is the software around the model that runs those steps, enforces the rules, and keeps the record.

The feature was a second opinion. When an agent finished, a separate model read what it had delivered and said whether it had done the task. If not, the agent got one try to fix it. I called it the output check.

And I had made it cheap in a way that felt clever. Every agent Cerevisor drives directly is asked to end its answer with one line: a confidence number between 0 and 1, plus one short sentence naming its main source of doubt. If that number came back at 0.8 or higher, the output check did not run. Why pay for a second opinion on work the author is sure about?

That question is the myth this piece is about.

Why a confidence score looks like a fair shortcut

I want to give the idea its due, because I shipped it. Checking everything costs money and time, and a person reviewing agent work has more output than attention. A number from the agent itself is the cheapest sorting signal there is. It arrives for free, on every run, in a tidy format.

The instruction Cerevisor gives is even careful about it: it tells the agent to use 0.90 or more only if it verified its output against the requirements, and 0.50 or below if it could not finish or had to guess.

The word that matters here is calibration. A source is calibrated when its stated confidence matches how often it turns out right: of all the times it says 80 percent, about 80 percent are correct. The whole shortcut rests on agents being calibrated about their own work. A note in the Cerevisor source says, in effect, that the newest large models are calibrated well enough for this. I wrote that down as an assumption and then built on it as if it were a measurement.

What studies of agent self-assessment found

Three pieces of research take that assumption apart, from two directions.

The first direction is the agent. A team led by Jean Kaddour asked AI agents to estimate their own chance of success before, during and after a task. It is a workshop paper from ICLR 2026, a machine-learning conference, so it has had lighter review than a journal article, and I am presenting it at that weight.

An older and more heavily reviewed study points the same way. Xiong and colleagues, at ICLR 2024, tested several ways of asking models for their confidence and concluded that models, “when verbalizing their confidence, tend to be overconfident.” The Kaddour paper adds a detail worth keeping: the framing that calibrated best was telling the agent to go and find bugs.

The second direction is the person. In Nature Machine Intelligence last year, Mark Steyvers and colleagues measured what they call the calibration gap: the difference between how confident people are in a model’s answer and how confident the model actually is. People overestimated the accuracy of answers that came with the model’s default explanation. And, in the paper’s words, “longer explanations increased user confidence, even when the extra length did not improve answer accuracy.”

Put those together. The agent rounds its odds up. Then its long, fluent summary rounds the reader up again.

A confident agent is not lying. It is doing what the study measured: sounding surer than its track record, to a reader built to believe it.

This is not about reviewers getting tired. A fresh, careful reader gets pulled too, because the pull comes from the writing.

What two weeks of real runs showed in Cerevisor

Here is what the output check did on my own machine, as recorded in the 3.7.1 release notes: “Over two weeks of real runs, that check was skipped more often than it ran: 82 of about 320 times because the delivered files were too long to read in full. None of its 6 rejections led to a correction.”

The notes do not break out how many skips came from an agent reporting high confidence, and I will not guess. What I can say is what the code did with each answer.

Then I found the part that stung. Cerevisor has a warning built for the exact case the research describes: an agent that reported high confidence and then had its work rejected by the check. The warning uses the same 0.8 line. So any answer confident enough to trigger it was also confident enough to skip the check that would have produced the rejection. Reading the 3.7.1 code for this piece, I could not find a path where that warning lights up. I had let the student grade the homework and also decide whether the teacher needed to see it.

In 3.7.1 the output check is off unless someone turns it on. The confidence line is still collected and still shown.

Read confidence as a doubt signal, never as a pass

A low number is worth acting on. An agent that says 0.4 and names what it could not confirm is handing over a lead. A high number is worth close to nothing as a reason to look away. It is the default register of a system that tends to round up.

That is why the check the release notes point to for work that matters is a reviewer card: a second agent that can only read, and whose verdict does not count unless it attaches evidence. It can be wrong too. But it is not the author, and it has to show something.

I am not alone in tidying this up. Across two Claude Code releases this past week, 2.1.288 on 2 October and 2.1.290 on 5 October, I counted four fixes for the tool saying something had succeeded when it had not, including a login that reported success when the credentials could not be saved. Software announcing its own success is an old habit. Agents just say it in better prose.

One question to ask of any agent setup this week

For anyone weighing a tool that runs several agents together, or already operating one, here is a concrete thing to do. Find every place where the agent’s own statement about its work switches off a check, shortens a review, or turns a row green. Ask the vendor, or read the settings. Then ask where the decision comes from when the agent says nothing.

In my own product the answer was one line of logic, and it had been sitting there since spring. I find that oddly reassuring. The fix for this kind of trust problem is rarely more vigilance from a tired person. It is moving one decision to a place where confidence cannot reach it.