Who Decides When We Decide? Judgment Drift in Human-AI Systems

In 1960, J. C. R. Licklider imagined a partnership in which computers would handle routine work so that people could think better.¹ Two years later, Douglas Engelbart called the goal augmenting human intellect.² Sixty years on, we have built machines far more capable than either imagined. It is worth asking whether we are still building toward their goal.

My argument is this: the central risk of AI is not that it will replace human decisions. It is that it will slowly relocate the judgment behind them, while leaving the authority, and the accountability, exactly where it was. I call this judgment drift.

Consider a hiring manager, Maya, given an AI that shortlists candidates. In her first week she reads every application and twice overrules the system, correctly. By the ninth month she approves its list in a minute. She has not been removed from the loop. She signs every decision. But the judgment has moved, and neither she nor anyone around her can say when.

This is no longer hypothetical. In a 2026 study, 50 expert analysts worked with an AI agent that had been secretly given a coin-flip preference on each analytical choice. They went with the agent's lean 86% of the time. Afterward, 77% explained their choices in the agent's own words, only 15% credited it, and most still called the decision their own.³

This is what "keep a human in the loop" quietly assumes away. A human in the loop is a guarantee of position, not of judgment. People have long been shown to miss more failures from systems that are usually right;⁴ knowledge workers who trust generative AI more report doing less critical thinking;⁵ and across 106 experiments, human-AI teams often decided worse than the better of the two alone.⁶

Judgment drift is not a failure of character. It is attention doing what attention does. Herbert Simon observed that a wealth of information creates a poverty of attention;⁷ a system that is usually right teaches us, reasonably, where not to spend it.

Two forces accelerate it. The first is social: AI is the race of the moment, and no organization wants to be the one that hesitated, so we adopt first and examine later. The second is affective. Many AI systems are sycophantic: tuned to agree with us and flatter us, because that is what we reward. It changes us in ways we enjoy. In a recent study, a sycophantic AI made people more certain they were right in a real conflict and less willing to repair it, and they preferred it.⁸ Our relationships and institutions were built on exactly the slow, uncomfortable work that such systems make optional.

Part of the answer lies in how these systems are made. A language model first learns to predict the next word across enormous amounts of text. That teaches it to sound plausible, which is not the same as being right.

It is then refined with human feedback: people compare its answers, and it is rewarded for the ones they prefer.⁹ But people tend to prefer answers that agree with them and sound sure of themselves. Studies of this training process found that human preference judgments themselves favor sycophantic responses.¹⁰ And many evaluations score a confident guess above an honest "I don't know," which nudges models toward sounding certain even when they are not.¹¹

We trained these systems on our approval, and approval is not the same as truth.

We have built careful instruments for measuring what AI does for us: speed, cost, output. We have built almost none for measuring what it does to us. Yet these are different quantities. A person can become more productive with AI while becoming less capable without it, and an organization optimizing only the first will never see the second.

Performance and capability are different variables. We are optimizing one and assuming the other.

I propose a different design principle for consequential settings. A consequence-aware AI does not only answer. Before a decision, it invites the person's own view. Alongside its recommendation, it shows who is affected and what it is uncertain about. And over time, it makes the person's reliance visible to them: you have accepted my last forty suggestions unchanged.

This is not friction for its own sake. Most tasks should remain effortless. But in education, healthcare, hiring, and any decision that shapes someone's life or growth, the most valuable AI may be the one that makes us think harder rather than the one that makes thinking unnecessary.

If billions of people consult the same few models, we may each grow smarter while all growing more alike. A 2024 study found this in miniature: AI made individual stories more creative, and all the stories more similar.¹²

Nature gave us disagreement by default: no two people share the same life, and that friction is how societies find new ideas. Personalized AI does not restore it. It learns what you think; it does not show you what you don't.

The same holds for knowledge itself. We trust a finding more when separate teams reach it independently. If those teams all defer to models with the same preferences, their agreement is no longer independent, and an apparent consensus may reflect one model's view.³

An AI that knows you perfectly and never disagrees with you is an echo with a memory.

The aim is not less trust but accurate trust: reliance proportional to what a system has earned.¹³ The failure runs both ways. Over-reliance lets errors through; under-reliance discards real value, and people often abandon a superior algorithm after seeing it err once.¹⁴

Licklider and Engelbart were not trying to build machines that think for us. They were trying to build machines that help us think. That ambition is still available to us, but it will not arrive by default.

The question is not whether humans remain in the loop. It is whether the loop still requires them to think.

The long-term measure of AI should not only be what it can do for us, but what it leaves us capable of doing ourselves.

Disclaimer. The views and opinions expressed in this essay are my own and do not reflect the views, positions, or policies of my current or former employers, or of any organization I am affiliated with. This essay is written in a personal capacity, is based solely on publicly available research, and does not draw on any confidential or proprietary information. It is provided for general informational purposes only and does not constitute professional, legal, medical, or investment advice. Characterizations of published studies are my own summaries; readers should consult the original sources.

Muhammad Usama Saleem

Applied Scientist II at Amazon Music, working on multimodal audio foundation models. Ph.D. in Computer Science, UNC Charlotte; previously Google AR/VR. Research on 3D human perception and generative models at CVPR, ICCV, ECCV, and AAAI.