Human-LLM Collaborative Reasoning Techniques

🧠 Free Recall ❓ Quiz 🔍 Cues 🔮 Simulation 🥷 ShadowBox 📐 Encode Cards

Human-LLM Collaborative Reasoning Techniques
The core design principle: LLM handles search/compute/synthesis; human handles novelty detection, anomaly flagging, and out-of-distribution insight.

Techniques
1. Structured Divergence Prompting
LLM generates a comprehensive "known solution space" for a problem
Human reviews not to validate, but to notice what's missing or feels wrong
Human flags gaps → LLM expands on those specific gaps
Rationale: Humans are better at sensing absence than exhaustively searching

Steps
Ask the LLM for a comprehensive map of known/standard solutions to your problem.
Read the output asking "what's missing?" rather than "is this correct?"
Flag gaps — categories that should exist but don't, or that feel underdeveloped.
Send those gaps back as targeted prompts for expansion.

Example. You ask: "How can a city reduce traffic congestion?" The LLM returns the standard menu — congestion pricing, transit investment, bike lanes, ride-sharing, remote work, road expansion. You notice nothing addresses why trips happen in the first place. You prompt: "Expand on solutions that reduce trip generation rather than manage existing trips." The LLM then explores mixed-use zoning, 15-minute neighborhoods, school assignment policy, and telemedicine.

eg. Novel applications of synthetic intelligence beyond chatbots

2. Synthetic Rubric Generation + Human Override
LLM generates evaluation criteria (rubrics) based on training data patterns
Human reviews rubrics for blind spots — criteria the LLM wouldn't know to include
Human adds novel criteria → LLM re-scores or re-ranks using the augmented rubric
This directly leverages synthetic rubrics as a scaffold, not a ceiling

Ask the LLM to generate evaluation criteria for the decision or artifact.
Review for blind spots — what would matter that an averaged training signal would miss?
Add domain-specific, contrarian, or long-horizon criteria.
Have the LLM re-score or re-rank options using the augmented rubric.

Example. Evaluating startup pitches. The LLM produces a rubric: market size, team experience, traction, defensibility, unit economics. You add two criteria the LLM wouldn't surface: "founder's psychological stamina under public failure" and "will this category still exist in five years, or is it riding a current trend?" The LLM re-evaluates three pitches; one that scored highest under the standard rubric drops considerably under the augmented one.

eg. Evaluating personal software utility

3. Hypothesis Lattice Exploration
LLM generates a tree/lattice of hypotheses from a seed question
Human navigates the lattice, pruning familiar branches and flagging unexpected ones
Novel branches get deep-dived by LLM
Human acts as anomaly detector, LLM as depth engine

Steps
Give the LLM a seed question and ask it to generate a branching tree of possible hypotheses, causes, or explanations.
Navigate the tree, pruning branches that feel obvious or well-trodden.
Star branches that feel surprising or under-examined.
Have the LLM deep-dive each starred branch.

Example. Seed: "Why are young men's social outcomes declining?" The LLM produces branches — economic, educational, cultural, technological, biological, structural. You prune the economic and educational branches as well-covered. You star "decline of male institutions" because it feels under-examined. The LLM deep-dives into fraternal orders, religious participation, mentorship structures, and same-sex friendship norms.

eg. SaaS terminal value under scrutiny

4. Red Team / Steel Man Loops
LLM steelmans the dominant view
Human challenges it with intuitions that feel true but aren't well-supported in literature
LLM searches for weak evidence supporting the human's intuition
Surfaces pre-paradigmatic ideas that lack citation density

Steps
Ask the LLM to make the strongest possible case for the dominant view.
Push back with an intuition you can't fully defend with sources.
Ask the LLM to find the weak or marginal evidence supporting your intuition.
Iterate — does the marginal evidence cohere into something?

Example. Consensus: "Lecture-based teaching is inferior to active learning." The LLM steelmans this with the standard meta-analyses. Your intuition: "But the most influential teachers I had were lecturers — the studies seem to miss something." The LLM searches for variance in lecture quality, narrative cognition, the role of expert demonstrations, and parasocial dimensions of learning. A thin but coherent literature emerges on "expert lecturer as cognitive scaffold" that complicates the simple consensus.

5. Cognitive Offloading with Checkpoints
Human defines the problem frame and constraints
LLM handles exhaustive search within that frame
Human reviews outputs at intervals, reframing when results feel "too expected"
Reframe triggers a new LLM search pass
Prevents LLM from optimizing into local minima of familiar solutions

Steps
Define the problem frame and what counts as a valid solution.
Let the LLM exhaustively search within that frame.
At intervals, ask: "Do these results feel too expected?"
If yes, reframe the problem and re-run.

Example. Frame: "Improve employee retention at a tech company." The LLM returns the expected list — comp, career development, manager training, flexibility. Checkpoint: too expected. Reframe: "What do people stay for that isn't compensation, and how do you make leaving non-financially costly?" The new pass surfaces mission-tied identity, irreplaceable peer relationships, mid-career skill specificity, and community formed through the job.

6. Contrastive Summarization
LLM summarizes what is well-established on a topic
Human reads for emotional or intuitive dissonance — things that feel incomplete
Those dissonances become new prompts
Human gut-check as a signal, not noise

Steps
Ask the LLM to summarize what is well-established on a topic.
Read slowly, noting where you feel uneasy or sense something incomplete.
Articulate each dissonance and feed it back as a prompt.
The LLM either resolves the dissonance or surfaces the gap.

Example. The LLM summarizes consensus on diet and longevity. You feel uneasy at "moderate alcohol consumption is associated with longevity" — something feels off. You articulate: "this looks like a selection effect — sick people stop drinking, making abstainers look unhealthy." You prompt this back. The LLM surfaces the recent literature on abstainer bias and revised meta-analyses showing no protective effect once that bias is controlled.

7. Weak Signal Amplification
Human brings a vague, half-formed idea
LLM finds the closest analogues in existing knowledge
Human assesses whether analogues capture the idea or distort it
If distorted, the distortion itself reveals what's novel
The gap between the idea and its best analogue = the novel contribution

Steps
Bring a vague, half-formed idea — phrase it imperfectly.
Ask the LLM for the closest existing concepts or frameworks.
Compare each analogue to your idea — where does the analogue feel wrong?
The mismatch is the novelty. Articulate what your idea has that the analogues don't.

Example. Half-formed idea: "Professional identity is starting to function like a religion." The LLM offers analogues: civic religion, secular religion, workism, Protestant work ethic. You note workism is closest, but it's about valuing work — your idea is about work replacing religion's structural functions (community, transcendence, moral framework, meaning at scale). That gap is the novel contribution, and it's now articulable in a way it wasn't before.

8. Iterative Boundary Testing
LLM defines the "known boundary" of a concept
Human proposes edge cases that feel like they should exist but don't appear in LLM output
LLM searches for thin evidence; human evaluates plausibility
Useful for identifying emerging phenomena not yet in training data

Steps
Ask the LLM to define the boundary of a concept — what it includes, what it excludes.
Propose edge cases that feel like they should exist but aren't appearing.
Ask the LLM to find any evidence, however thin, of those edge cases.
Evaluate plausibility — these may be emerging phenomena not yet well-documented.

Example. Concept: "parasocial relationships." The LLM defines this as one-sided emotional bonds with media figures — celebrities, streamers, podcasters. You propose edge cases: parasocial bonds with AI assistants, with deceased people reconstructed via media, with one's past self through old journals. The LLM finds thin but real literatures on each, with the AI-companion case rapidly expanding — a clear instance of an emerging phenomenon outstripping its concept.

The most powerful loop is: LLM maps the known → Human senses the edge → LLM probes the edge → repeat. The human is not validating LLM output — they are using it as a contrast medium to make their own novel intuitions visible.

Unstarred

Dunk (exclude from study)

Human-LLM Collaborative Reasoning Techniques

Review Questions

Todos

Sensemaker