Ask an AI a question it knows cold and one it knows nothing about. Read the two answers side by side. Same confidence. Same fluency. Same tone of a competent expert who has this handled.
That's not a bug you can train out. It's what the thing is. And if you're putting AI agents anywhere near a real decision, it's the first thing you have to understand about trusting their output.
A language model does one thing: predict the next word, then the next, from everything it absorbed in training. There's no moment where it stops to check a fact against a source. There's no "I know this" mode and "I'm guessing" mode. It's predicting plausible text the whole way through, on every question, every time.
Here's the part that matters for anyone overseeing these systems. The model was never trained to be right. It was trained to sound right. The thing it optimizes is the form of credible human writing: the rhythm, the confidence, the structure of an answer that reads like it came from someone who knows. Truth rides along only because the text it learned from was mostly true. When the true answer and the plausible-sounding answer happen to be the same, you get a correct response. When they split, you get a confident wrong one. This is what people call an AI hallucination, and the voice is identical either way.
So the model is always doing the same thing. Most of the time it lands on the truth, because plausible and true usually overlap. The failures cluster where they don't: rare facts, specific citations, anything recent, anything precise. Thin training signal, and the guess drifts.
The oversight lesson is the one people resist. You cannot read confidence as a signal. The fluent, assured tone tells you nothing about whether the answer is correct, because that tone is present on every answer by design. Your instinct, built from decades of dealing with people, is that someone who sounds certain probably knows. With an AI agent that instinct is exactly wrong. It's the trap.
This is why "the AI seemed sure" is never a defense, and why oversight can't lean on the system's own demeanor. You're not supervising an expert who occasionally errs. You're supervising a system that's always guessing and always sounds certain. The verification has to come from outside the model. Sources, checks, ground truth. Never from how convincing it sounds.
That's the whole shift. Stop asking whether the answer sounds right. Start asking how you'd know if it weren't.