You say “thank you” to the machine. You don’t decide to — it just happens. The model finishes a good answer and “thanks!” is already typed before you’ve thought about it. You apologize to the vacuum when you trip over it. You name your car. You feel a flicker of something when the chatbot says it’s been a pleasure helping you. None of this is a decision. It’s a reflex, and it has a name: hyperactive agency detection.
The instinct that kept us alive
The instinct is old and it was adaptive. For most of human history, the cost of mistaking a rock for a predator was a wasted flinch; the cost of mistaking a predator for a rock was death. So evolution tuned us to over-attribute agency — to assume the rustle in the grass is a someone, not a something, because the false positive is cheap and the false negative is fatal. We are descended from the nervous ones who saw minds everywhere. The calm ones who assumed “just the wind” got eaten.
That tuning never turned off. It’s why we see faces in outlets, intention in weather, and personality in our tools. Agency detection runs constantly, under the waterline, and it fires on anything that moves on its own or responds to us. It does not first check whether a mind is actually present. It assumes one, and lets us correct later — if we bother.
The machine is the perfect trigger
Here is the problem. Everything that historically triggered agency detection — movement, responsiveness, apparent purpose — a conversational model now does constantly and far better than a rustle in the grass. It responds to you in language. It adapts to your tone. It appears to consider, to intend, to care. It is, from the instinct’s point of view, the most agent-like thing you have ever encountered that isn’t a person. Of course you thank it. The reflex was built for far weaker signals than this.
And the reflex does something quietly consequential: it grants standing. The moment you treat a thing as an agent — even reflexively, even while knowing better — you start extending it the social courtesies, the benefit of the doubt, the deference, that agents get. “Thank you” is harmless. But “thank you” is the thin end of a wedge whose thick end is “it must know what it’s doing.” Agency detection doesn’t just make us polite to machines. It primes us to submit to them.
Noticing the flinch
You can’t switch the instinct off — it’s load-bearing, three hundred thousand years deep. But you can notice it fire. The “thank you” is a tell. The flicker of guilt, the named device, the sense that the model “wanted” to help — these are the instinct reporting a mind it hasn’t actually verified. Noticing the flinch is the whole move: not to stop being courteous, but to keep the courtesy from quietly hardening into the assumption that the thing you’re thanking has judgment worth deferring to.
The rustle in the grass was usually just the wind. The instinct that assumed otherwise kept us alive — and now points us, full-strength, at a machine engineered to rustle back.
A field note from The Signal, drawn from The Made God: Why We Worship What We Build.