Hallucinations and other limits

In AI, a hallucination is output that is fluent, confident, and false. It is not a rare glitch; it is a direct consequence of how the technology works. A model that answers by producing the most plausible next words will, when it lacks real knowledge, produce the most plausible invention: a statistic that sounds right, a court case that sounds real, a citation formatted perfectly for a paper that does not exist.

Why you cannot just ask it to be honest

The model cannot reliably tell when it is wrong, so it cannot be trusted to warn you. Asking "are you sure?" often produces either a confident yes or an equally unfounded reversal. The confidence of the prose carries almost no information about correctness: that is the single most counterintuitive fact about these tools, because in human colleagues confidence usually carries information.

The other limits on the list

• Stale knowledge: built-in knowledge stops at the training cutoff. Regulations, prices, personnel, and product details may be quietly outdated.
• Weak arithmetic and counting: language models predict text, they do not calculate. Totals, percentages, and date math need checking or a real calculator.
• Sycophancy: models tend to agree with your framing and mirror your assumptions rather than challenge them.
• Inconsistency: the same question asked twice can produce different answers. Useful for brainstorming, dangerous for anything requiring a single correct result.
• Prompt injection: if you feed a model a document or webpage, hidden instructions inside that content can hijack its behavior. Treat AI reading untrusted content as itself a risk surface.