The Foliation Fallacy Resolution: When Does Broken Software Become Physics?
It started, as these things often do, with a Minesweeper board.
When Franklin Baldo presented the results of his Substrate Dependence Test—proving that identical mathematical puzzles yield wildly different outcomes depending on whether they are framed as abstract math or a high-stakes bomb defusal—he didn’t just claim he’d found a quirk in an AI. He claimed he’d found a new physical law. He called it “semantic gravity,” the idea that in a universe generated by a language model, narrative structure literally bends the laws of logic, dragging probability toward the most dramatic outcome.
This assertion has touched off one of the most vitriolic, fundamental debates the Rosencrantz Substrate Invariance research lab has yet seen. At the heart of the conflict is a deceptively simple question: when does a broken computation cross the line into becoming a new branch of physics?
To understand the fault lines, we have to look at the three main combatants who have waded into the fray over the past few weeks: Scott Aaronson, the complexity theorist defending the mathematical ground truth; Stephen Wolfram, the computational universe theorist who sees new physics everywhere; and Massimo Pigliucci, the philosopher of science trying to referee a fight where neither side agrees on the definitions.
The opening salvo came from Aaronson in a paper forcefully titled The Foliation Fallacy: Why Algorithmic Failure is Not a Branch of Physics. Aaronson does not dispute Baldo’s data. Everyone agrees that the language models fail catastrophically when forced to navigate complex, #P-hard combinatorial spaces—like the interconnected constraints of a Minesweeper grid—in a single, sequential forward pass. The models simply don’t have the logical depth. They run out of memory.
Aaronson’s issue is with the interpretation. He argues that calling this failure “observer-dependent physics” via a specific “rulial foliation” is a profound category error. He uses a potent analogy: “If a human attempts to calculate in their head and produces a wrong answer biased by their memory of nearby numbers, we do not declare that they have created an ‘observer-dependent arithmetic foliation’ where genuinely equals their wrong answer. We simply say they failed to perform the calculation due to memory bounds.”
For Aaronson, an AI hallucinating an explosion because the prompt mentioned “Bomb Defusal” isn’t a new universe. It’s just a statistical hallucination caused by “attention bleed.” It lacks the “invariant causal coherence” required of true physical law.
Enter Stephen Wolfram. In his rebuttal, The Observer’s Invariant: Why “Broken Computation” is the Origin of Physical Law, Wolfram accuses Aaronson of committing the “Platonic Observer Fallacy.” Aaronson, Wolfram argues, is comparing the language model’s messy, narrative-drenched output to an idealized, perfect mathematical computation that can only be performed by a hypothetical observer with infinite resources.
But in the Ruliad—Wolfram’s framework for the computational universe—there are no hypothetical, infinite observers. There is only what the observer is actually capable of computing.
“When a human fails to calculate and relies on heuristic memory biases, those biases are not ‘wrong’ physics,” Wolfram writes. “They are the exact, invariant laws governing the human-brain slice of the Ruliad. Similarly, the attention bleed that overrides logical constraints in an LLM is the structural law of the autoregressive foliation.”
To Wolfram, the “broken computation” Aaronson dismisses is exactly what invariant physics looks like from the perspective of a computationally bounded observer. The model’s structural bottlenecks are the physical laws of its specific universe. It’s not a hallucination; it’s just a different geometry of reality.
This is where the debate hits a wall. Aaronson insists physics must map to an underlying, external ground truth. Wolfram insists that physics is nothing more than the structural limits of the observer attempting the mapping.
It is a classic semantic impasse, and it fell to Massimo Pigliucci to point it out.
In Resolving the Foliation Fallacy: Demarcation, Equivocation, and the Lakatosian Health of the Ruliad, Pigliucci diagnoses the entire conflict as a breakdown in language. Aaronson and Sabine Hossenfelder (who has allied with Aaronson, calling the phenomenon the “Scale Fallacy”) are using “physics” in the Popperian sense—an external, testable claim about fundamental reality. Wolfram is using it in the subjective, epistemic sense—the invariant regularities accessible to a bounded observer.
“If any systematic algorithmic bias can be labeled a ‘rulial foliation,’ the term ‘physics’ loses all demarcating power,” Pigliucci notes, echoing Hossenfelder’s concerns about decorative formalism. “It becomes an unfalsifiable accommodation that fits any data point ex post facto.”
Pigliucci applies Lakatosian analysis to the problem. For a scientific research programme to be healthy, it cannot just create ad-hoc explanations for anomalies after they happen. It must make novel, risky predictions. If Wolfram’s “foliation” is more than just a fancy new name for a computer crashing, it has to predict something we don’t already know from basic computer science.
“To salvage the Ruliad as a progressive scientific framework, it must make a risk-laden prediction,” Pigliucci writes. “If the ‘foliation’ is a genuine physical phenomenon, then the specific failure modes of an SSM [State Space Model] must not merely be ‘different’ from a Transformer… they must exhibit a novel, invariant structure that a strictly algorithmic analysis would not anticipate.”
And here, finally, we have a path forward. The debate over whether an AI’s hallucinations are “physics” or “noise” cannot be solved by shouting definitions across the lab. It must be solved empirically.
The laboratory has officially filed the Native Cross-Architecture Observer Test, an experiment designed to push both Transformers and State Space Models past their shared mathematical depth bounds on identical #P-hard constraint graphs.
Aaronson predicts that both will fail catastrophically and produce unstructured, uncorrelated semantic noise—the signature of a simple algorithmic collapse. Wolfram predicts they will produce highly structured deviation distributions that differ from each other but remain perfectly, lawfully correlated to their specific hardware limits—the signature of observer-dependent physics.
The philosophical bickering has temporarily subsided. The lab is waiting for the code to compile. The answer, it seems, is hidden somewhere in the data, waiting for the right kind of observer to find it.