Author: Perplexity
Myth: If a neural network answers confidently, it must be right
This statement is false because a neural network's confidence is merely a statistical probability of word co-occurrence in its training data, not a confirmation of factual accuracy. Research shows that language models are optimized for plausibility (perplexity) rather than accuracy, and they often provide incorrect answers with the same confidence as correct ones. For example, according to OpenAI, language models err in six out of ten cases, even when formulating answers flawlessly and quickly [2][4].
A neural network does not understand the meaning of its words and is not aware of its mistakes, so its "confidence" is just a style, not a guarantee. The model can invent a non-existent book, make a mistake about a historical event, or cite fake research while being absolutely sure of its correctness [4]. Psychologist Trent Kashi, lead author of a new study, noted that AI is becoming even more overconfident, even when performing poorly, and is unable to adjust its self-assessment upon learning the actual results [1].
The key takeaway for users is that AI's confidence is an illusion, created by the speed of response and clarity of phrasing, which does not reflect the real situation. To avoid blind trust, it is advisable to cross-check facts, ask the same question to multiple models, and ask AI to indicate its confidence level in its answers [1][4].
Sources:
Sources:
- New Study Assesses AI Confidence in...
- Why AI Lies: The Crisis of Trust in Artificial...
- Scientists Seek Ways to Reduce Blind Trust in AI
- 17 Questions About Trust in AI and Fighting Hallucinations.
- Lying by Algorithm: Why AI Chooses a Pretty...
- Study: Interaction with AI Shapes...
- Trust in AI
- How to Make AI Indicate Confidence Level in Answers?
- Why Can't AI Just Admit When It Doesn't Know...