Category Errors of AI/Agents

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

The "intelligence" error. We talk about AI systems as more or less "intelligent" along a single axis, often borrowing the human IQ frame. But intelligence in humans is a tangled bundle of capacities—perception, memory, motivation, social modeling, embodied skill, metacognition—that evolved together and constrain each other. Current AI systems have wildly uneven profiles: superhuman at some things, subhuman at others, and nothing at many things humans don't even notice they're doing (like knowing they have a body, or that time has passed). Ranking them on a one-dimensional "smartness" scale treats a multi-dimensional thing as scalar. The implication: benchmarks that produce a single score, and discourse about whether AI has "passed" humans, are often measuring something that doesn't exist as a unified quantity.
The agent/tool error, in both directions. Some people treat language models as pure tools—stochastic parrots, autocomplete, no different in kind from a calculator. Others treat them as nascent agents with goals, preferences, and inner lives. Both might be category errors, because the systems may not fit cleanly into either bucket we inherited. A model that produces goal-directed-looking behavior in some contexts and pure pattern-completion in others isn't well-described by either frame. The implication: safety arguments that depend on "it's just a tool" and safety arguments that depend on "it's an agent with goals" can both miss, because they're arguing about which familiar box to put a thing that may need a new box.
The "understanding" error. Asking "does the model really understand?" often smuggles in a presupposition that understanding is one definite thing, present or absent, like pregnancy. But understanding in humans is itself a graded, multi-faceted phenomenon—you understand a poem differently than you understand a proof. Treating AI understanding as a yes/no question forces a binary onto something that may need a richer typology. Searle's Chinese Room, for instance, is sometimes accused of this: it assumes "real" understanding is a definite property the room either has or lacks, when the more interesting question might be what kinds of understanding are at stake.
The consciousness/moral status error. This one cuts both ways. Confidently asserting AI systems are conscious, or confidently asserting they aren't, may both be category errors if our concept of consciousness was built around biological organisms with continuous existence, embodiment, and evolutionary history. The honest position might be that we don't yet know whether the question even applies in its usual form. The implication: dismissing AI welfare concerns as obviously silly, and asserting AI suffering as obvious, may both be premature—not because we lack evidence, but because we may lack the right concepts.
The alignment-as-control error. A lot of alignment discourse treats the problem as: how do we make sure the AI does what we want? This frames AI as a tool to be aimed. But if these systems are something more like new kinds of minds (or new kinds of something, even if not minds), then "control" may be the wrong frame in the same way that "controlling" a human collaborator is the wrong frame for getting good work from them. You'd want trust, shared understanding, legitimate authority structures. Conversely, treating alignment as if it were primarily a relational/ethical problem when the system in front of you really is more tool-like is also an error. The category may genuinely shift across systems and contexts.
The economic-substitution error. Discussions of "AI taking jobs" often model AI as a worker-shaped thing that slots into existing job categories. But the more interesting historical pattern is that transformative technologies reorganize work into new categories rather than substituting one-for-one. Asking "will AI replace radiologists?" may be the same shape of question as "will cars replace horses in the cavalry?"—technically yes, but the question dissolves once cavalry itself stops being a category.
The benchmarking error. Treating "performance on task X" as evidence of a capacity assumes that humans and AI systems use the same underlying capacity to perform X. When a model passes the bar exam, we infer lawyerly competence. But the model may be solving the test through a route that doesn't generalize the way human bar-passing does. We're treating evidence-of-X-in-humans as evidence-of-X-in-AI, when X may not be a unified capacity that travels across substrates.
The implications, taken together, are something like this: a lot of AI discourse is loud, confident, and arguing about answers to questions that may be the wrong shape. This produces three kinds of waste—policy built on misframed problems, research effort aimed at malformed targets, and public discourse that polarizes around false binaries (sentient/not, dangerous/safe, real intelligence/mere mimicry). The most productive move, I think, is the Rylean one: before working harder on the answers, pause and ask whether the question has the right shape. New phenomena sometimes need new categories, and forcing them into old ones can look like rigor while actually being confusion.

Unstarred

Dunk (exclude from study)

Category Errors of AI/Agents

Review Questions

Todos

Sensemaker