In June 2023, a New York attorney filed a court brief packed with case law citations — cases that never existed. ChatGPT had fabricated them, and the lawyer never bothered to check. The Mata v. Avianca incident became a symbol of something researchers have studied for decades: the misplaced trust we put in automated systems.
📖 Read more: AI Drug Discovery: New Drugs in 18 Months
Automation Bias: When We Trust Machines Blindly
Automation bias is the tendency to accept suggestions from an automated system without critical evaluation, even when they contradict our own observations. The term was established by Mosier and Skitka in 1996 while studying pilots who ignored their own instincts to follow autopilot indications.
The phenomenon manifests in two ways: as errors of omission — we fail to notice mistakes because we trust the machine — and as errors of commission — we follow incorrect instructions without questioning them. In aviation, maritime operations, and medicine, the examples are deeply troubling.
The Boeing 737 MAX tragedy is perhaps the most dramatic example. The MCAS (Maneuvering Characteristics Augmentation System) repeatedly pushed the aircraft's nose down based on data from a faulty sensor. The pilots, trained to trust automated systems, fought a machine that overrode them — until two planes crashed, killing 346 people.
Overtrust vs. Undertrust
Researchers Raja Parasuraman and Victor Riley, in their landmark 1997 paper, categorized trust problems in automation into three types: misuse (excessive reliance), disuse (rejecting useful tools), and abuse (automating where it's not needed). The problem isn't whether we trust or don't trust — it's that we consistently misjudge when to trust.
📖 Read more: AI Journalism: Robots Writing the News
The Trust Paradox: In a University of Pennsylvania experiment (Dietvorst, Simmons & Massey, 2015), participants preferred their own judgment over an algorithm — even when the algorithm was demonstrably more accurate. All it took was seeing it make a single error. What the researchers called algorithm aversion proves just as dangerous as blind faith.
On the other hand, research by Logg, Minson, and Moore (2019, Management Science) revealed that in certain contexts, people show algorithm appreciation — assigning greater reliability to algorithmic predictions than to human ones, even when they don't understand how they work. Both tendencies operate in the same person. Context, training, and emotional state tip the balance.
Real-World Examples: What Goes Wrong
The COMPAS Criminal Justice AI
The COMPAS system (Correctional Offender Management Profiling for Alternative Sanctions) is used across the US to predict recidivism risk. ProPublica's 2016 investigation found the system was nearly twice as likely to falsely label a Black defendant as high-risk compared to a white defendant. Despite this, judges in multiple states continued relying on its “objective” scores — a textbook case of automation bias in practice.
Tesla Autopilot and the Illusion of Autonomy
The name “Autopilot” creates an illusion of full self-driving that the technology doesn't support. The NHTSA (National Highway Traffic Safety Administration) has investigated dozens of fatal crashes where drivers removed their hands from the wheel, trusting a Level 2 system as if it were Level 5. The language we use to describe AI systems — “intelligence,” “autopilot,” “assistant” — isn't innocent. It shapes our expectations.
"People tend to treat automated systems as though they have human-like properties — a tendency psychologists call anthropomorphism. This leads to unwarranted trust, especially when systems 'speak' in natural language."
📖 Read more: Artificial Consciousness: Can an AI Think?
— Parasuraman & Manzey, Human Factors (2010)What the Data Shows
According to the Stanford HAI AI Index 2025, public concern about artificial intelligence is steadily increasing worldwide. In the US, Pew Research reports that 52% of Americans express more concern than excitement about AI's role in daily life — a 14-point increase in just two years. In Europe, the AI Act (in force since August 2024) reflects the institutional response, categorizing AI systems by risk level.
The survey data tells only half the story. While people say they're worried, their daily behavior simultaneously increases AI interaction — over 100 million weekly ChatGPT users prove it. Trust works like a dimmer, not a switch, adjusting with each interaction.
📖 Read more: Biological Computers: Neurons Instead of Transistors
Calibrated Trust
Neither blind acceptance nor outright rejection works. Researchers propose what they call calibrated trust — the ability to trust a system exactly to the degree its actual performance justifies.
Three factors make the difference:
- Transparency: Explainable AI systems (XAI) — like the LIME and SHAP tools — allow users to understand why an algorithm made a particular decision, rather than accepting a “black box.”
- Training: Understanding an AI system's limitations is as important as learning to use it. A doctor needs to know when an AI diagnostic tool is reliable — and when it isn't.
- Feedback: Users who regularly receive performance data about AI calibrate their trust better than those who simply use the outputs without verification.
The DARPA XAI Program: DARPA (Defense Advanced Research Projects Agency) launched a $75 million program in 2017 to create AI systems that explain their decisions. Six years later, explainability remains one of the field's biggest open problems — especially for large language models composed of billions of parameters.
Why the Problem Is Getting Worse
The proliferation of large language models (LLMs) added a new dimension to the trust problem. Unlike traditional AI systems — which provided numbers, search results, or risk scores — LLMs “speak” in natural language, use persuasive tone, and produce answers that feel authoritative even when entirely fabricated.
📖 Read more: Neuromorphic Chips: Chips That Think Like a Brain
What researchers call "hallucination" isn't a bug — it's a fundamental property of probabilistic language models. An LLM doesn't “know” anything. It predicts likely word sequences. But the fluency and confidence of its text creates an impression of expertise in the reader.
The risk is compounded by a version of the Dunning-Kruger effect: the less someone knows about a topic, the less able they are to judge whether the AI's answer is correct. The attorney who filed the fake citations wasn't lazy — he lacked the tools to critically evaluate a tool that spoke in flawless legal language.
The Path Forward
We've reached a turning point with AI. We can't go back to life without algorithms, but we can't sleepwalk into the future either. The real challenge isn't building better AI — it's training ourselves to use it wisely.
Some steps are already underway: the European Union now requires impact assessments for high-risk AI systems, while companies like Anthropic and Google are experimenting with “confidence calibration” mechanisms — systems that express their degree of certainty rather than presenting every answer as irrefutable.
But the final line of defense remains human judgment. Not as an adversary to technology, but as its complement. We don't need less AI. We need to learn when to trust it — and when to walk away.