LLMs & Generative AI · 2.2

🎭 Why AI Makes Things Up

Hallucinations, and why a confident AI can be confidently wrong⏱ ~2 min

The Fundamental Limitation

⚠ WarningAn LLM generates the most *plausible-sounding* next words — not the most *true* ones. When it doesn't know something, it doesn't stop; it produces text that looks right. This is called a hallucination: confident, fluent, and completely made up. It's not lying (that requires intent) — it's doing exactly what it was built to do, which is generate plausible text.

Why This Happens

  • •The model optimizes for plausible-sounding text, not verified facts
  • •It has no built-in concept of 'I don't know' unless specifically trained to say it
  • •Gaps in training data get filled with confident-sounding guesses
  • •It can invent citations, court cases, statistics, and quotes that never existed
★ FactIn 2023, two lawyers were sanctioned by a US court after submitting a legal brief full of fake case citations that ChatGPT had invented. The cases sounded completely real — proper names, plausible quotes — but none existed. The lesson: fluency is not accuracy.

How to Protect Yourself

  1. 1.Verify facts independently — treat AI output as a confident intern's draft, not gospel
  2. 2.Ask for sources — then check them — and confirm the sources actually exist and say what's claimed
  3. 3.Be extra skeptical of specifics — exact numbers, dates, names, and quotes are prime hallucination territory
  4. 4.Use AI for drafting and brainstorming, not as your only source of truth on important things
  5. 5.Cross-check anything that matters — medical, legal, financial, or academic claims especially
🔒 SecurityThis limitation is also a security issue: never paste secrets, passwords, or private personal data into a public AI tool. Your input may be used to improve the model, could be seen during safety review, and leaves your control the moment you hit send. Treat a chatbot like a public forum.
🧠Quick Checkfirst try = +5 XP

An AI confidently gives you a fake but real-sounding citation. This is called…

🎮 Practice what you learned
🎣
Phishing Spotter
Spot AI-generated phishing before you click
→🃏
Attack Sorter
Sort AI-era attacks into the right categories
→
⭐ 0 XP🔥 0 days
Open the full interactive AI & Cybersecurity module →