How To Evaluate AI Tools

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

The Compounding Test
A way to evaluate any AI tool — bought, built, or in the wild — by the structural properties that determine whether it can actually improve over time. Tools with these properties compound. Tools without them plateau, no matter the effort, talent, or budget behind them.

1. Defined target There is a concrete, written definition of what "good output" means for this tool's task. "Generated code passes the existing test suite" is defined. "Helps users be more productive" is not.

2. Objective scoring That definition can be checked without subjective judgment. Rank what's available:
A: Machine-checkable — tests, ground-truth matching, deterministic outcomes
B: Structured human rubric with measured inter-rater agreement
C: Ad-hoc human review, satisfaction scores, NPS
D: No real measurement
A or B means the tool has a real improvement path. C or D means it doesn't.

3. Fast signal The gap between an output and a quality signal is short. Seconds-to-hours = the system can learn quickly. Days-to-weeks = glacial. No regular signal at all = no learning.

4. Closed loop The quality signal actually feeds back into changes — model updates, prompt revisions, training data, retrieval tuning, eval-driven regression catches. A signal nobody acts on isn't a loop, it's a dashboard.

Reading the result
All four present → the tool will compound. Expect meaningful improvement over months.
Missing #4 → measurable but stuck. It works as well as it ever will.
Missing #3 → improvement is too slow to matter on your timeline.
Missing #1 or #2 → no real improvement path exists. Any apparent gains are noise or selection effects dressed up as progress.

Independent verification Whatever the tool, maintain your own frozen eval set on representative data and re-run it on a fixed cadence. This is how you separate real improvement from model swaps, prompt drift, and marketing. Without your own eval, you can't tell whether the tool you're using is getting better, getting worse, or just changing.

Why this works This is the structural reason some AI categories race ahead while others stall. Codegen compounded fast because "good" is concrete (the code runs, the tests pass), the signal is instant, and it feeds straight back into training and evals. Categories where "good" is subjective, slow to measure, or never flows back into the system tend to look impressive at launch and stay roughly there.

Unstarred

Dunk (exclude from study)

How To Evaluate AI Tools

Review Questions

Todos

Sensemaker