Why 'Cognitive Debt' From AI Coding Agents Is a Security Problem
A widely-shared talk from Notion design engineer Geoffrey Litt argues that as agents write more code, understanding it becomes the real bottleneck — and for security teams, that understanding gap is where review controls quietly fail.
Key Takeaways
- Geoffrey Litt (Notion, ex-Ink & Switch) argues the bottleneck in agent-assisted development has shifted from writing code to understanding it well enough to keep steering the project.
- He frames this using Margaret Storey's 'cognitive debt': skipping the work of building a mental model costs nothing immediately, but the debt compounds until reviewers have effectively 'lost the plot.'
- For security teams, that gap is where deterministic controls fall short — a plausible-looking but wrong permission check or a silently weakened default survives review because no human retains the context to challenge it.
- Litt's proposed fixes (structured explainers, interactive 'micro-worlds', shared team mental models) don't replace scanning or blind review; they're aimed at keeping humans capable of meaningfully auditing what agents produce.
At the AI Engineer conference, Notion design engineer and former Ink & Switch researcher Geoffrey Litt gave a talk he later expanded into an essay, "Understanding is the new bottleneck". Simon Willison flagged it as one of the sharper framings he's seen for a problem engineering teams are now hitting daily: as coding agents take on larger, more autonomous changes, the constraint shifts from *writing* code to *understanding* it well enough to keep participating in the project's direction.
The cognitive debt problem
Litt borrows the term cognitive debt from Margaret Storey. Like technical debt, skipping the work of building a working mental model of a system costs nothing in the moment — but it compounds. Eventually, in Litt's words, "the humans involved may have simply lost the plot." His point isn't that agents can't be trusted to write correct code; verification is something agents already do reasonably well. It's that *participation* — redirecting a project, catching a bad abstraction early, or pushing back credibly on a design choice — requires a working set of concepts in your head that only comes from sustained understanding, not from skimming diffs.
Why this belongs on a security team's radar too
Code review is itself a security control, and it depends on the reviewer actually holding the mental model needed to spot when something is wrong. Static analysis, dependency scanning, and blind QA passes catch a class of defects, but the subtler failures — a permission check that looks right but isn't, a config default flipped as a side effect of an unrelated change, a trust boundary quietly moved during a large refactor — are exactly what a reviewer who has 'lost the plot' waves through. As agentic tools take on bigger, multi-file, multi-service changes, the review question shifts from "did someone check this line" to "does anyone still hold the context needed to know this line is wrong." That's a real gap, not a hypothetical one, and it sits upstream of incidents that later get labelled as supply-chain or logic-flaw issues.
- Auth or authorization logic silently weakened across a large agent-driven refactor, with no single reviewer holding the full before/after picture
- A configuration default changed as an incidental side effect of an unrelated agent task, missed because it wasn't the stated goal of the change
- A dependency or integration swapped by an agent without a human registering that the trust boundary moved
What Litt proposes — and how it maps to engineering practice
Litt's essay offers three techniques for keeping understanding intact as agents do more of the work: explanations (structured explainer documents that give background and an intuitive overview before showing the raw diff), micro-worlds (interactive debuggers and simulations that let a person build understanding by exploring behaviour rather than passively reading code), and shared spaces (collaborative environments where a team builds a mutual mental model together, rather than each person reconstructing it alone). None of these replace deterministic review, scanning, or a build-gate — they're aimed at keeping the human capable of meaningfully auditing what the agent produced in the first place.
The point was always to augment, not just automate.
— Geoffrey Litt
For organisations already doing AI governance work — mapping AI-assisted development against a framework like ISO 42001 — this gives the 'human oversight' requirement teeth. Understanding isn't a soft aspiration; it's a control that can be designed for, staffed for, and tested, in the same way access control or input validation are.
Frequently Asked Questions
What is "cognitive debt" in AI-assisted development?
A term Geoffrey Litt attributes to Margaret Storey, describing what happens when developers let their understanding of a codebase drift from what an AI agent has actually implemented. It costs nothing at first, but compounds until reviewers can no longer meaningfully evaluate or direct changes.
Does this mean AI coding agents are insecure?
No — Litt's argument is about human review capacity, not agent capability. Agents are generally reasonably good at producing verifiable output; the risk is that humans stop holding enough context to catch the subtler defects that slip past automated checks.
How can engineering teams keep oversight as agents write more code?
Litt suggests structured explainer documents before raw diffs, interactive tools ('micro-worlds') for building hands-on intuition, and shared team spaces for building a common mental model — used alongside, not instead of, existing review and security gates.
Sources
- 1Understand to participate — Simon Willison's Weblog
- 2Understanding is the new bottleneck — geoffreylitt.com
- 3Understand to participate — Simon Willison