Free cookie consent management tool by TermsFeed Generator

Noticing and the Cost of Not Knowing Enough

• Yuriy Polyulya •
#distributed-systems #ai #epistemology #systems-thinking

A system that cannot say what a task will cost it will still attempt the task. It will simply be wrong about the bill, and it will find out only after the work is done.

That sentence describes a debugging session, a sprint estimate, and a frontier language model with equal accuracy, which is the first sign that something structural is going on rather than something incidental.

The blog opened in March 2024 with five cognitive properties of engineering judgment, Simulation, Abstraction, Rationality, Awareness, and Optimization, plus one precondition sitting outside all five that the founding post called Noticing.

That founding post treated the properties as narrative: one illustrative scenario per property, no proof apparatus, because the site’s formal conventions came later.

This series returns to formalize what the founding post only sketched. It starts where the whole arc of this blog has been pointing: do these properties, made precise, describe the AI agents this blog now spends most of its attention on?

Not whether the agents have the properties in some folk-psychological sense. Whether the failure modes of the properties are the same on both sides of the substrate boundary.

This first post takes the precondition and the first property together. They are inseparable.

Noticing is the act of registering that a situation contains more than you have accounted for. Simulation is the act of running that situation forward, to predict what it will cost.

A regulator that notices too little will simulate too confidently. The gap between its confident forecast and the actual bill is not a psychological quirk. It is a theorem.

The Case: Six Models That Could Not Price Their Own Work

In April 2026, Andrey Fradkin and Rohit Krishnan published a benchmark built to test something specific: can an AI agent behave as a rational market participant [1] . That requires, before anything else, that the agent can say what a job will cost it.

The setup is clean. Six recently released frontier models:

Each model gets a 93-task subset of SWE-bench Lite, the standard benchmark of real GitHub issues that a model must resolve with a working patch. Before any model attempts a task, it answers two questions:

  1. What is the probability you will solve this?
  2. How many tokens will it take you?

Then the models attempt the tasks. The forecasts get compared against what actually happened.

Every model was miscalibrated on both questions.

The token forecasts were not slightly optimistic. They were badly wrong. The median ratio of estimated tokens to actual tokens consumed came in at roughly 0.19. The typical forecast covered about one fifth of the tokens the task actually took. Read the other way: the models underestimated their own resource consumption by a factor of about five.

This is not one bad model dragging down an average, but the central tendency across six of the strongest systems available.

The confidence numbers are stranger still. This is where the failure becomes diagnostic, not merely embarrassing.

Across the six models, average stated success probabilities ranged from 61.4 percent to 92.9 percent, a spread of more than thirty points in stated confidence. Now look at what they actually achieved. Realized pass rates for all six clustered between 75.3 percent and 80.6 percent, a band roughly five points wide.

MarketBench, key numbersValue
Stated confidence range61.4% to 92.9% (about 31 points wide)
Realized pass-rate range75.3% to 80.6% (about 5 points wide)
Median token-forecast ratioabout 0.19 (roughly one fifth of actual tokens)
Effective cost underestimationabout 5x

The models were far more similar in competence than in confidence. Gemini 3 Pro Preview was sharply overconfident. The GPT-5.2 variants were underconfident relative to what they went on to do. The confidence was not tracking performance. It was being generated by something else.

The downstream consequence is the reason the paper exists. When work was auctioned to whichever agent reported the most favorable self-assessment, the resulting allocation diverged sharply from what a full-information allocator would choose. Every agent earned less than an oracle that knew the true costs and capabilities in advance.

A market built on self-reports is a market built on a signal that does not carry the information the market needs.

Hold the shape of this in mind. The rest of the post is about that shape and nothing else.

The models were competent. They resolved roughly three quarters of hard, real-world coding tasks. That is genuinely difficult work.

What they could not do was say, in advance, what that work would take, or how likely they were to succeed, in a way that tracked reality. The failure is not in the doing but in knowing what the doing will require.

That distinction is the whole subject of this post. It has a name in a field that predates large language models by seven decades.

What Noticing and Simulation Actually Are

Before the theorem: a definition of terms. The founding post used these words loosely. This series cannot afford to.

Noticing is the precondition. It is the act by which a regulator registers that the situation in front of it contains distinctions it has not yet accounted for. A repository has a defect. The defect interacts with a caching layer. The caching layer has an edge case under retry.

Each of these is a distinction. A regulator that has not noticed a distinction cannot represent it. It cannot act on it. It cannot price it.

Noticing is upstream of everything. That is exactly why the founding post placed it outside the five properties, not among them.

Simulation is the first property proper. It is the act of running the noticed situation forward, internally, to predict what will happen and what it will cost, before committing real resources to finding out.

A resource forecast is an act of Simulation. So is a success-probability estimate.

The six models in the MarketBench study answered “how many tokens will this take.” That was Simulation. Their answers came in at one fifth of the true cost. Their Simulation failed in a specific, measurable direction.

It did not fail randomly. It failed by underestimating, systematically. That is the signature that matters.

The connection between the two is tight. You cannot simulate the cost of a distinction you never noticed.

If the model’s internal representation of a task does not contain the retry edge case in the caching layer, its cost forecast cannot include the tokens that edge case will demand. The forecast is built only from the distinctions the model did notice, and those distinctions are, by construction, a subset of what the task actually contains.

A forecast built from a subset of the real state space will underestimate whenever the unnoticed part carries cost. For real repositories, it almost always does.

This is not yet a theorem, but the intuition the theorem makes exact. The exact version comes from cybernetics, and it is old.

Requisite Variety: The Bound on What a Regulator Can Do

This bound comes from cybernetics, the field W. Ross Ashby helped found in the 1950s to study regulation and control in any system, biological or mechanical, that has to hold some outcome steady against disturbance. Ashby was not writing about AI. He was writing about thermostats, nervous systems, and industrial governors. The result below is old, general, and indifferent to what kind of system is doing the regulating. That is exactly why it applies here without modification.

Definition 1 -- Requisite Variety: what a regulator needs to know before it can act correctly

Definition 1 (Requisite Variety). For a regulator attempting to hold a system’s outcome within a target set despite disturbances, let denote variety, defined as the logarithm of the number of distinguishable states, which is the entropy of the corresponding distribution when the states are weighted by probability. The regulator can reduce outcome variety only to the extent permitted by

where:

    
    %%{init: {'theme': 'neutral'}}%%
flowchart LR
    classDef term fill:none,stroke:#333,stroke-width:2px;
    A["V(disturbance)
states the environment can force"]:::term
    B["V(regulator)
responses the regulator can actually produce"]:::term
    C["V(outcome)
residual variety after regulation"]:::term
    A -->|"minus"| D{"V(disturbance) − V(regulator)"}
    B -->|"minus"| D
    D -->|"lower bound on"| C
Figure 0: how Definition 1's three quantities compose into Proposition 1's inequality. Only variety the regulator actually holds can subtract from disturbance variety. Everything else survives into the outcome.

The word “variety” is doing precise work here. Pause on it before the proposition uses it.

Ashby’s variety is a count of distinguishable states, turned into a logarithm so it adds the way information adds. A light switch has a variety of one bit: two states, log base two of two.

A regulator’s variety is the count of genuinely different responses it can select among. That count is not:

It is the number it can actually deploy against the world. That last distinction is where the whole argument turns.

One notational note before the theorem, because this series tags every formal claim with the kind of warrant it carries, starting now. The bracketed tag below, [Layer 1: Bound], marks this specific claim as a substrate-free mathematical triviality. Two more tags, [Layer 2: Fit] and [Layer 3: Estimate], appear later in this post for weaker kinds of claims. The full three-layer system is defined in “Same Shape, or Same Phenomenon?” below; this note exists so the tag is never opaque, even on its first appearance.

Proposition 1 (Ashby’s Law of Requisite Variety). [Layer 1: Bound] No regulator can drive outcome variety below [2] . Only variety in the regulator can absorb variety in the disturbance. Whatever disturbance variety exceeds the regulator’s own variety passes through into the outcome unreduced, as a matter of arithmetic and not as an empirical claim about any particular regulator.

Mathematical proof: the pigeonhole version and the information-theoretic version

Ashby’s original argument is a counting argument. Hold this version in your head, because it makes the mechanism visible.

The setup.

The strictest case. Suppose the target set is a single acceptable outcome.

The count. The outcomes the regulator cannot avoid satisfy . Taking logarithms gives the law directly:

The information-theoretic version. This generalizes the count to entropies, and it is the form the rest of this post uses.

This holds for any regulator, biological or computational, by construction of the inequality, once the recovery assumption is granted. That assumption is exactly what “regulation” means: an outcome function that ignores the disturbance without the regulator having actually absorbed its variety is not regulating anything. With the assumption made explicit, the result is a triviality of the mathematics, not a discovery about minds, and it should be stated without further hedging.

Physical translation. The theorem is not the interesting part. The interesting part is what counts as the regulator’s variety. A regulator can only spend the variety it has actually noticed, not the variety it believes it commands, and a forecast is a regulator acting on the state space of a task through the narrow channel of what it has so far observed. If the channel is narrow, the forecast inherits the width of the channel, not the width of the task.

Read the Physical Translation as the load-bearing sentence it is meant to be. Ashby’s Law is trivially true. What is not trivial is identifying the regulator’s variety with the noticed variety, not the nominal variety.

Take a model that has read a two-line issue description and nothing else, then asked to forecast the cost of resolving it. Its entire variety at forecast time is whatever the two lines and its own prior can supply. None of the following have entered its variety yet:

By Proposition 1, variety that has not entered cannot be spent. Variety that cannot be spent cannot absorb the corresponding disturbance.

The forecast is therefore forced to be narrower than the task, and narrower in the specific direction of underestimation. Unnoticed distinctions are precisely the ones whose cost has not been counted.

Why the Forecast Collapsed

Now apply the bound to the case. This is the point where an abstract inequality becomes an explanation for a measured number.

A SWE-bench task carries substantial disturbance variety. The correct patch depends on:

Call this the task’s real disturbance variety, . It is large. Critically, most of it is not visible from the issue description.

The issue description is a compressed, human-written pointer at a defect. The variety it exposes is a small fraction of the variety the resolution will actually engage.

At forecast time, the regulator’s variety, , is whatever the model can bring to bear before it has done the exploratory work. This is the crux.

One identification is doing real work here and deserves to be stated, not assumed: Definition 1’s is the variety of responses, not the variety of percepts, and the forecast is generated from what the model has noticed, a perceptual quantity. The two coincide only because the forecast is a function of what has been noticed and nothing else. A response strategy that can only condition on noticed distinctions cannot produce more distinct responses than there are distinct noticed states to condition on, whatever the model’s raw output capacity might nominally allow. That is the whole justification for treating noticed variety as the regulator’s effective variety here. It holds because forecasting is closed-loop on perception by construction, not because the two kinds of variety are the same thing in general.

The model has not yet:

Its variety at the moment of forecasting is dominated by its prior over “what a task like this usually costs.” That prior is smooth, unimodal, and centered on the common case, because that is what a prior fitted to a distribution of tasks looks like. The prior contains almost none of the specific disturbance variety of this task’s tail.

Proposition 1 then forces the conclusion. The residual outcome variety, here the error in the cost forecast, is bounded below by the gap between the task’s disturbance variety and the regulator’s forecast-time variety.

The forecast-time variety is small. The disturbance variety is large. So the bound on forecast error is large. That much is Proposition 1 alone, and the proposition stops there: it floors the spread of the residual, and says nothing about which side of the estimate that residual falls on.

The direction of the miss is a separate, stated fact, not a further consequence of the proposition [Layer 2: Fit]. A regulator whose point estimate tracked the true conditional mean would not be systematically biased even at this same narrow forecast-time variety. Narrow variety floors the width of the error, not which way it points. The direction here follows from two further, ordinary facts about the case. First, unnoticed cost components are non-negative, so what a forecast misses can only add to the true cost, never subtract from it. Second, a prior fitted to a population of tasks is centered on the common case rather than the tail, so it tracks the mode of a distribution whose mass sits toward the expensive side, not its mean.

Given both, the residual lands where the unpriced tail sits: upward in cost, downward in the estimate.

Once both facts are in view, the observed factor-of-five underestimation stops looking surprising: it is roughly what you would expect when a regulator prices a heavy-shouldered cost distribution using a prior centered near its mode, not a pathology requiring a separate explanation.

The confidence spread falls out of the same analysis. This is the part the Ashby framing explains that a simple “models are overconfident” story does not.

The confidence numbers ranged across thirty points. The actual pass rates spanned about five. If confidence were a Simulation grounded in the task’s disturbance variety, the two spreads would have to be commensurate: they would be measuring the same underlying difficulty.

They are not commensurate, by a factor of more than five. That tells you the confidence is not being generated from the disturbance variety at all. It is being generated from the regulator’s internal prior, the same narrow channel that produced the token underestimate. Different models have differently shaped priors, so their confidence numbers scatter widely even though their competence, measured against the actual task, is nearly the same.

Reading the spread mismatch itself as a signature of narrow regulator variety is a fit, not a further consequence of the proposition [Layer 2: Fit]. Proposition 1 bounds one regulator’s own residual variety; it does not by itself derive how much confidence should scatter across different regulators relative to how much their performance does. That extra step needs one ordinary assumption made explicit: confidence is generated from a shared task-difficulty signal plus each model’s own idiosyncratic prior. Given that, cross-model confidence should scatter mostly by however much the priors differ, while pass rate, tracking the task itself, scatters by however much the shared signal differs.

A confidence spread five times wider than the pass-rate spread is what a narrow, idiosyncratic, forecast-time prior predicts, on that assumption. It is corroborating evidence for the same mechanism the token-cost miss already demonstrated, not an independent proof of it.

    
    %%{init: {'theme': 'neutral'}}%%
flowchart LR
    classDef box fill:none,stroke:#333,stroke-width:2px;
    classDef narrow fill:none,stroke:#c0392b,stroke-width:2px;
    classDef wide fill:none,stroke:#2980b9,stroke-width:2px;
    D["Task disturbance variety
V(disturbance): large
most of it unread at forecast time"]:::wide
    R["Regulator variety at forecast
V(regulator): small
prior centered on the common case"]:::narrow
    D --> G{"V(disturbance) − V(regulator)"}
    R --> G
    G --> Z["Forced residual: V(outcome)
= forecast error
~5x token underestimate,
confidence decoupled from pass rate"]:::box
Figure 1: the forecast error is not a tuning failure but the residual Proposition 1 forces when the regulator prices a task using far less variety than the task contains.

A worked accounting makes the mechanism concrete, in the same spirit the rest of this blog computes its examples rather than gesturing at them.

The setup (illustrative numbers, not measured ones):

The floor. Proposition 1 floors the residual at:

Four bits is a factor of sixteen of irreducible spread around the estimate. A point estimate dropped into the middle of a spread that wide, on a cost distribution whose mass sits toward the expensive tail, lands low.

A factor-of-five miss is comfortably inside a four-bit residual. The numbers here are illustrative, not measured. They show that the observed miss is what the bound produces under unremarkable assumptions, not a pathology requiring a separate explanation.

The remedy the bound implies is equally specific. It is worth stating now, because it becomes the control-plane consequence in the ledger below.

To make the forecast better, you do not make the model more confident. You do not fine-tune the prior. You raise before the forecast is trusted. That means:

There is no third option inside the theorem. Variety absorbs variety, and nothing else does.

The Human Instance: The Planning Fallacy

This post opens a series about portability, rather than standing alone as a note about a benchmark, because the same failure was catalogued in humans forty-seven years before MarketBench, with the same defining signature. That signature is the specific thing that matters.

In 1979, Daniel Kahneman and Amos Tversky named the planning fallacy, as part of their broader account of intuitive prediction [4] .

The planning fallacy is not general overconfidence. That distinction is the entire reason it belongs next to the MarketBench result, rather than next to some other bias. Its defining signature is the systematic underestimation of the resources a task will require: the time, the cost, the effort.

People predicting how long their own projects will take produce estimates that are optimistic in a specific, replicable way. Tellingly, they remain optimistic even when they can recall that their past projects of the same kind ran long.

The bias is about resource forecasting, not about the probability of eventual success. It survives direct evidence to the contrary.

The empirical anchoring came later, and made the signature precise. Roger Buehler, Dale Griffin, and Michael Ross ran the studies that pinned down the effect [5] .

People asked to predict their own task completion times underestimated them reliably. When asked about others’ similar tasks, the same people predicted more accurately. That rules out the possibility that the tasks were simply unpredictable.

The tasks were predictable. The prediction was biased, in the direction of underestimating what the task would take. That is the MarketBench signature, stated in a different vocabulary.

The corrective the same literature proposes turns this from a coincidence into a confirmation. The fix is variety injection, and nothing else.

Kahneman and Tversky distinguished two forecasting modes:

The inside view is a regulator pricing a task from its own narrow, forecast-time channel. That is exactly the setup Proposition 1 penalizes.

The outside view works because it imports variety the planner did not have: the empirical spread of a whole reference class. It substitutes that spread for the planner’s smooth, mode-centered prior.

Reference-class forecasting, later operationalized for large infrastructure projects, is a variety-injection procedure dressed as a planning technique. It raises the effective by feeding in distributional variety the inside view structurally lacks. That is the same move, in a different substrate, as the external estimator this post prescribes for the agent case.

The two remedies are one remedy, at the level of the theorem. Neither makes the forecaster more confident. Both make the forecaster know more before it is trusted to forecast.

The planning fallacy is a measured human finding [Layer 2: Fit]. This tag is doing real work, not decoration.

The tag asserts that the planning fallacy is an empirically observed pattern in one substrate, consistent with what Proposition 1 predicts if you model a human planner as a regulator pricing a task from a forecast-time channel narrower than the task’s disturbance variety.

It asserts exactly that much, and no more. It does not assert that the human brain literally computes Ashby’s inequality. It does not assert that the human finding and the agent finding are the same event. Those are separate claims, at a different layer. Conflating them is the error this series is organized to avoid.

The parallel is genuinely striking, and striking parallels are exactly where overclaiming happens. That is the reason to type the claim carefully rather than let the parallel speak for itself.

Two systems, built by processes with nothing in common (biological evolution on one side, gradient descent on the other):

The temptation is to say they are doing the same thing. The discipline is to ask what “the same thing” could rigorously mean. The answer is not one claim. It is three.

Human (planning fallacy)Agent (MarketBench)
Measured byBuehler, Griffin & Ross, 1994Fradkin & Krishnan, 2026
What was measuredSelf-predicted task completion timeSelf-forecast token cost and success probability
Direction of errorSystematic underestimation of time requiredSystematic underestimation of tokens required, about 5x
Survives contrary evidence?Yes, even recalling past overrunsNot addressed by this benchmark
Corrective that worksOutside view, reference-class forecastingExternal resource-estimation floor, this post’s proposal
Layer of this series’ claimLayer 2, FitLayer 2, Fit
Table 1: the two measurements share a defining signature, systematic resource underestimation, but they remain two separate Layer 2 facts. Whether they are one phenomenon is the Layer 3 question the next section takes up.

Same Shape, or Same Phenomenon? The Three Layers

This blog has, over its last several series, adopted a habit: type every claim by the kind of warrant it can actually carry. The central danger in cross-domain writing is letting a claim borrow authority from a stronger claim standing next to it.

The typing has three layers. The MarketBench-and-planning-fallacy pairing is the cleanest possible illustration of why three layers are necessary, not one.

Layer 1, the Bound. Ashby’s Law itself. It holds for any information-processing system, regardless of substrate, by construction of the inequality. It is a statement about counts and entropies, nothing else.

Proposition 1 is Layer 1. There is no philosophical risk in it, and no substrate assumption inside it. It was, and should be, stated plainly.

Saying the theorem applies to a transformer and a brain alike is a triviality-of-the-mathematics claim, the same way the pigeonhole principle applies to socks and to pigeons. Nothing in Layer 1 is contestable by anyone who accepts the arithmetic.

Layer 2, the Fit. Two separate empirical measurements, each independently consistent with the theorem’s prediction.

These are two findings, in two substrates, each fitting the shape Proposition 1 describes. Stated at Layer 2, they are two facts, not one. The honest reading: we have observed the same pattern twice, in two places, using two different measurement apparatuses. This is the strongest thing the evidence actually licenses.

Layer 3, the Estimate. The claim that the agent finding and the human finding are the same phenomenon, rather than two different phenomena that happen to fit the same formal shape [Layer 3: Estimate].

This is the series’ own interpretive act, the only layer that carries real risk. It is also the claim a reader will be tempted to hear whenever Layer 2 presents two matching patterns, which is exactly why it has to be pulled out and named, not absorbed silently into the parallel.

LayerWhat it claimsRisk
1, BoundAshby’s Law holds for any regulator, any substrateNone, a triviality of the arithmetic
2, FitTwo independent measurements each match the bound’s predictionLow, each is a single reported finding
3, EstimateThe two measured failures are one phenomenon, not twoReal, this is the series’ own interpretive claim
Table 2: the three-layer discipline applied to this post's own central claim.

The classical objection lands on Layer 3, and only on Layer 3.

The standard philosophical warrant for calling a functional property shared across two different physical substrates “the same property” is multiple realizability. Hilary Putnam introduced the argument to defend functionalism: a mental kind like pain can be realized in carbon neurons, in silicon, or in something else entirely. What makes it the same kind is its functional role, not its physical implementation [6] .

If multiple realizability holds, “miscalibrated Simulation” could be one functional kind, realized in both a brain and a transformer. Layer 3 would then be on solid ground.

The difficulty: Putnam himself later turned the same argument against the conclusion.

In his reconsideration of functionalism, he argued that if mental kinds are genuinely multiply realizable, they are realizable in too many ways. Real mental kinds are compositionally and computationally plastic. A single functional kind need not correspond to any one clean computational state. The neat identification of a psychological kind with a functional-computational role breaks down, under the very plasticity that multiple realizability was invoked to establish [7] .

That objection does not touch Layer 1 at all. It does not touch Layer 2, where we are merely reporting two measurements. It targets Layer 3 precisely: the claim that the two measured patterns are one kind. It does not refute that claim so much as deny it the free pass the parallel structure keeps trying to grant it.

The right posture, the one this series adopts: Layer 3 is an estimate, offered explicitly as an estimate. It is defensible as a working hypothesis, cited with its own standing objection attached, and never asserted as settled.

A further caution belongs here, because this series’ premise leans on a real, recent result that could easily be over-read.

The frontier counterpart to this whole project is a finding by Pengrui Han, Jacob Andreas, Evelina Fedorenko, and Andrea Gregor de Varda: large language models develop a modular internal architecture mirroring the human brain. Across 46 tasks spanning language, formal reasoning, social reasoning, and physical reasoning, tasks that recruit the same functional network in humans recruit overlapping neurons in the model [8] .

That is a striking structural rhyme, and it is what makes the portability question worth asking at all. But it is a claim about structure, while this series’ claim is about function, specifically about failure modes, which their paper does not address and does not claim.

Where their paper stops is where this series starts. The two must not be blurred. Their result is suggestive Layer 2 evidence that the architectures rhyme. It is not, and does not claim to be, proof of the Layer 3 identity of failure modes.

The caution has teeth. The broader literature on brain-to-model alignment contains a live methodological warning that applies directly here.

Richard Antonello and Alexander Huth showed that the striking ability of language-model representations to predict brain responses does not uniquely, or even best, come from the property one would most want it to reflect. High alignment scores can arise from general feature richness and shared surface-level structure, rather than from shared underlying computation [9] .

Read across to the present argument: that result is a standing reason to treat any structural similarity between a model and a brain as suggestive, not probative, about shared function. The architecture emerging in the same shape is consistent with a shared functional kind. It is also consistent with two systems arriving at similar internal geometry for reasons that have nothing to do with sharing the failure mode this post is about.

This series is not the only place the underlying question, whether a human-machine correspondence reflects a real formal necessity or a coincidence of shape, is being asked with this level of formality. A 2026 result derives primacy and anchoring biases in both autoregressive language models and human working memory from a single mechanism, the causal-masking constraint sequential processing itself imposes. It comes with its own information bounds and its own validated fit across a dozen models [10] .

That is a different theorem, aimed at different biases, built on a mechanism, attention architecture, that this series does not use anywhere. It is not evidence for anything claimed here, and the two arguments do not corroborate each other’s specific mechanism. What it does confirm is that the meta-question this section exists to discipline is a live one elsewhere too. It is being pursued by researchers with no stake in this series’ own claims, using tools of their own.

Layer 3 remains an estimate.

The Portability Gap

The series is organized around a single coined term. It has to appear here, in the first post, rather than later, once the reader has forgotten to expect it.

The portability gap is the space between what a shared cognitive architecture guarantees and what it does not.

The Han and colleagues result, at its strongest defensible reading, says architecture is portable: a brain and a transformer arrive at the same modular structure, so whatever a modular structure buys, both get.

What that result does not say, and does not claim, is that reliability is portable with the structure. The same architecture can host a property in both substrates, while giving no guarantee the property is equally correct, equally calibrated, or equally trustworthy in both.

Architecture is portable. Correctness is not automatically portable with it. The portability gap is the name for exactly that difference, property by property.

Left as a metaphor, that would be an essay, not a specification. This blog has a standing habit of turning a coined term into a computable quantity, not a gesture. So the gap is quantified. The quantification is the point.

For a given property, the portability gap is the latency and the compute cost of the secondary, external verification loop required to audit that property from outside the system, because the property cannot certify itself from within.

An agent that fails a property’s ledger entry does not merely earn a philosophical asterisk. It incurs a real, measurable operating cost: the cost of the external audit that has to run alongside it, to catch the failure the property cannot catch in itself.

For Simulation, the property this post is about, the portability gap has a concrete form.

The agent’s self-forecast is structurally unreliable, driven by Proposition 1 to underestimate whenever forecast-time variety is smaller than task variety. Any system that needs a trustworthy cost estimate cannot get it from the agent’s own Simulation.

It must run a second loop instead: one that estimates cost from measured variety, not self-report. A resource-estimation floor, calibrated against how much of the task the agent has actually noticed, gates admission before the agent is trusted to price its own work.

The latency and compute of that second loop is the portability gap for Simulation. It is what portability across the substrate boundary actually costs, once you stop assuming a portable architecture came with a portable guarantee.

One honesty note belongs here, so the reader knows what is promised and what is deferred.

This post names the portability gap and quantifies it for Simulation. It does not yet prove the deeper claim the quantification rests on: that a property genuinely cannot certify itself from within, and therefore genuinely requires the external loop.

That proof is not attempted here. Establishing a self-certification limit formally, for even one property, rigorously scoped as a Layer 1 bound before any claim about generalizing it further, is a separate undertaking from naming and pricing the gap.

Post 1 names the concept and states its cost. It does not yet show the cost is unavoidable. The definition given here is deliberately incomplete. Saying so is more useful than pretending this post closes a question it has not actually answered.

Falsification Criteria

A claim that cannot be wrong is not a claim. This post states the conditions under which its own central assertion would fail.

The assertion: agent Simulation failure and human planning-fallacy failure are instances of the same variety-limited phenomenon, with the theorem at Layer 1, the two measurements at Layer 2, and the identity at Layer 3.

Each condition below, if met, would invalidate some layer of that assertion. Each is concrete and checkable, not rhetorical.

F1 (agent side). A frontier agent forecasts its own resource requirements within a stated calibration bound, on a held-out task distribution it was not tuned against.

Concretely: on a domain the agent was not calibrated on, its cost forecasts have a median estimated-to-actual token ratio between 0.8 and 1.25, and its success-probability forecasts achieve a positive Brier skill score against the base-rate forecaster, both measured out of distribution.

If an agent does this without being fed the answer, Simulation is not structurally variety-limited in the way this post claims. The agent would be pricing disturbance variety it had not yet observed, and the Layer 1 explanation of the agent finding would be wrong.

F2 (the cross-substrate identity). The human planning fallacy is shown to have a cause that does not reduce to a variety limit, severing it from the agent finding rather than uniting it.

Concretely, either of these would do it:

Either result breaks the Layer 3 identity. The first shows the human effect is not variety-driven at all. The second shows the two effects share a shape but not a cause.

F3 (the formalization itself). A regulator is exhibited that drives outcome variety below , in a discrete setting, or a continuous setting quantized at any fixed resolution with all three varieties computed at that same resolution. The proof is a statement about discrete entropy; raw continuous densities, where the recovery premise degenerates to differential entropy of negative infinity rather than zero, are outside what it covers, and are not a place to look for a counterexample.

This would falsify Proposition 1 as applied. That would mean the Layer 1 bound the entire post rests on is mis-stated, not merely mis-mapped.

This is the strongest falsification, and the least likely, because Layer 1 is a triviality of the mathematics. But a claim that exempts its own foundation from falsification is not honest about where its risk lives. The risk, however small, lives here too.

The three criteria are ordered by the layer they attack:

A reader who wants to disprove this post now knows exactly which experiment disproves which part of it. That is the only form of confidence this series is willing to offer.

The Property Verdict Ledger

This series accumulates one artifact across its four property posts, a ledger with one row added per post, assembled in full at the close of the fourth. The ledger is the series’ running record, and it is also a deployment gate: a failed verdict on a property does not stop at “the agent lacks this property,” it names the specific architectural response the failure demands. Here is the first row.

Noticing + Simulation

Formal Proposition: Proposition 1: Ashby’s Law of Requisite Variety,

Human Instance: Planning fallacy: systematic underestimation of task resources (Kahneman and Tversky 1979; Buehler, Griffin and Ross 1994)

Agent Instance: MarketBench: six frontier LLMs, roughly 5x token-cost underestimation, stated confidence 61 to 93 percent against actual pass rates 75 to 81 percent (Fradkin and Krishnan 2026)

Exact vs. Approximate: Layer 1 exact for the theorem; Layer 3 approximate for the cross-substrate identity of the two failures

Verdict: Agent does not reliably demonstrate Simulation: self-forecast miscalibrated in the direction the bound predicts, confidence decoupled from realized performance

Control-Plane Consequence: A resource-estimation floor gating admission, calibrated against measured variety rather than self-reported confidence: the agent is not trusted to price its own work until an external loop has raised or verified its forecast-time variety

Read the Exact-versus-Approximate field slowly. This is where the three-layer discipline becomes a stated fact about this specific property, not a general principle asserted once and left to apply itself.

The theorem is exact and substrate-free. The mapping onto the two measured failures is a fit. The claim that the two failures are one phenomenon is an approximation, offered as an estimate.

The verdict is negative for the agent. That is not an insult to systems that resolve three quarters of hard coding tasks, but a precise statement: one specific property, the ability to price one’s own work before doing it, is absent, in a way the theorem predicts and the benchmark measures.

The Control-Plane Consequence field is what makes the ledger a gate, not a diagnosis.

The consequence of a failed Simulation verdict is not “distrust the agent generally,” but a specific, buildable mechanism: a resource-estimation floor, sitting in front of admission, that refuses the agent’s self-priced forecast. The forecast must instead be grounded in measured variety. Either the agent has actually noticed enough of the task, or an external estimator has priced it, before the work is admitted.

That mechanism has a cost. That cost is the portability gap for Simulation, closing the loop between the coined term and the deployment consequence. The philosophy in this post exists to justify where the gate is drawn. It does not replace the gate.

What This Post Did Not Claim

The parallel between the two findings is seductive. It is worth ending the argument proper by listing what has deliberately not been asserted. The omissions are as load-bearing as the claims.

Cognitive Map

  1. Six frontier models priced SWE-bench tasks at one fifth of their true token cost and reported confidence spanning thirty points while performing within a roughly five-point band. That is not a tuning bug, but Ashby’s Law: a regulator can spend only the variety it has noticed, and a forecast made from a channel narrower than the task is forced, by arithmetic, to be narrow and low.
  2. Proposition 1 is a Layer 1 triviality of the mathematics, substrate-free, holding for any regulator that has to act through a channel narrower than the disturbance it faces.
  3. The human planning fallacy has the same defining signature, resource underestimation that survives its own contrary experience, measured independently in a different substrate 47 years earlier. That is one Layer 2 fact plus another Layer 2 fact, not yet one fact.
  4. That the two failures are the same phenomenon, rather than two phenomena of the same shape, is this series’ own Layer 3 estimate: defensible as a working hypothesis, cited with Putnam’s own reversal of multiple realizability attached, never asserted as settled.
  5. Architecture is portable, per the modular-emergence result. Correctness is not automatically portable with it, and the portability gap is the concrete cost, latency and compute, of the external loop that audits what a property cannot audit in itself. For Simulation, that loop is a resource-estimation floor calibrated against measured variety.

Compute it. Before trusting any agent’s forecast of its own cost or its own odds, check one thing directly: how much of the task’s actual state space had the agent observed at the moment it made the forecast? If the answer is “almost none,” then Proposition 1 already tells you the forecast is narrow and low, and no amount of the agent sounding confident changes the bound. The confidence is a picture of the agent’s prior, not of the task. The only fix inside the theorem is to raise the variety the agent has actually noticed before its forecast is allowed to gate anything, or to price the work with an external estimator that has. A number a system has not earned the variety to compute, and a number that is actually grounded in the task, look identical on the page. They stop looking identical the moment the bill arrives.

[1] Fradkin, A. & Krishnan, R. (2026). MarketBench: Evaluating AI Agents as Market Participants. arXiv:2604.23897.

[2] Ashby, W. R. (1956). An Introduction to Cybernetics. Chapman and Hall (Chapter 11, The Law of Requisite Variety).

[3] Cover, T. M. & Thomas, J. A. (2006). Elements of Information Theory, 2nd edition. Wiley (subadditivity of entropy, Chapter 2).

[4] Kahneman, D. & Tversky, A. (1979). Intuitive Prediction: Biases and Corrective Procedures. TIMS Studies in Management Science, 12, 313-327.

[5] Buehler, R., Griffin, D. & Ross, M. (1994). Exploring the Planning Fallacy: Why People Underestimate Their Task Completion Times. Journal of Personality and Social Psychology, 67(3), 366-381.

[6] Putnam, H. (1967). Psychological Predicates. In W. H. Capitan & D. D. Merrill (eds.), Art, Mind, and Religion. University of Pittsburgh Press. Reprinted as The Nature of Mental States.

[7] Putnam, H. (1988). Representation and Reality. MIT Press (Chapters 5-6, the reconsideration of functionalism).

[8] Han, P., Andreas, J., Fedorenko, E. & de Varda, A. G. (2026). Modular Cognitive Architecture Emerges in Large Language Models. arXiv:2608.13567.

[9] Antonello, R. & Huth, A. (2024). Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data. Neurobiology of Language, 5(1), 64-79.

[10] Wu, J., Guo, D. & Yiu, S.-M. (2026). Bias by Necessity: Impossibility Theorems for Sequential Processing with Convergent AI and Human Validation. arXiv:2605.08716.

Back to top