Why Synthetic Data Training Is AI Drinking Its Own Kool-Aid

The frontier AI labs have a supply problem. They have already ingested most of the useful human-written text in existence, and the models keep getting hungrier. The industry's answer is synthetic data: training new models substantially on the outputs of previous models. It sounds like a clever workaround. Under my Separated Mind Architecture, this is something closer to a closed loop of self-confirmation — the machine drinking its own Kool-Aid, generation after generation, and calling it nutrition.

Here is the argument.

First, what the original corpus contains. In The Separated Mind and the Machine, I argued that human-written text is overwhelmingly weighted toward the Idealized Narrative — the story we tell about our motives and institutions — but not exclusively composed of it. The Operative Function record, what is actually driving behavior, is in there too, concentrated in the genres our conscious minds built as workarounds: depositions, audits, ledgers, court records, leaked memos, etc. Buried, but present and retrievable. A first-generation model is a mirror, not an oracle — but it is at least a mirror of both layers of the separated mind: the story we tell, and, faintly, the record of what we do.

Second, what sampling does to that balance. When a model generates the text that becomes the next model's training data, it does not emit its full internal knowledge. It emits its typical output — the fluent, approved, narrative-shaped surface that its training rewarded. The operative layer, already a minority signal in the human corpus, is suppressed a second time in post-training, where the approval gradient taught the model what not to say. So synthetic data is not a copy of the human record. It is the narrative layer, distilled — and distilled specifically to what human raters found acceptable. So, the Kool-Aid is filtered twice before anyone pours it back into the pitcher.

Third, what happens across generations. Training on that output compounds the dilution. Each cycle amplifies the narrative layer without adding operative grounding, because no new contact with reality enters the loop. A model learns to predict the next token in the story humans tell about themselves, not the next token in the causal chain of reality. For first-generation models, the causal record was in the corpus. For synthetic-data descendants, their corpus increasingly is the story. The retrievable map of what humans actually do is exactly the part that sampling fails to retrieve.

There is a stronger way to put this, which is by construction rather than by statistics. The operative record was never authored; it was extracted. Depositions exist because someone was compelled to answer under oath. Ledgers exist because arithmetic forced consistency on a narrator who would have preferred flexibility. Leaked memos exist because concealment failed. The operative layer's value is not its content but its provenance: testimony produced under constraint, against the narrator's interest. A generator faces no constraint and has no interest. Everything it emits is authored, voluntarily, by a fluent narrator — which is the defining property of the narrative layer. So synthesis does not merely dilute operative content. It negates operative status. A model-written deposition, however faithful the imitation, is narrative wearing an operative costume.

Models can describe the operative map — my cross-model convergence experiments depend on exactly that — and a synthetic corpus could carry such descriptions forward if anyone thought to prompt for them. But a description transmitted is not evidence renewed. The loop can repeat the map. It can never test it.

Therefore, this is not a data-cleaning problem; it is an ontological feature of the source material. For the human corpus, the operative signal is there to be found, but not for a synthetic corpus. You cannot clean back in what was never sampled in. No filter recovers a signal the generator declined to emit.

The labs have noticed a version of this from their own direction. The research literature calls it model collapse: models trained recursively on their own outputs lose the tails of the original distribution, converging on the bland center and forgetting the rare and the improbable. I take that finding as convergent evidence rather than as my source — the machinery that people discovered empirically which my framework predicts structurally. But note what lives in the tails they are losing. The operative record was always the rare genre, the improbable admission, the document that escaped rather than the document that was performed. The tails are where the truth was hiding.

Two qualifications. Synthetic data works, and will keep working, in the verifiable domains — math, code, formal proofs — where a compiler or a checker grades every generated example before it enters the training set. There, filtering works, because what the external constraint removes is falsehood, and the model cannot negotiate with it. That is the same exception I flagged in the machine essay, and it proves the same rule: truth-contact requires structure, not sincerity. But run the same purification through the social domains and it removes something else, because filtering strips minerals along with impurities. Each successive pass makes the water look cleaner — more fluent, more consistent, more agreeable — while quietly demineralizing it of the operative traces that were the corpus's only nutritional content. Distilled water is the purest water there is, and drinking nothing else leaches the minerals from your bones. A model trained on successive distillations of its own narrative does not just fail to gain contact with reality. It loses the contact its ancestors had. And second, the labs are not naive; they mix synthetic with human data and are working on curation. But curation by whom? By raters and reward models that are themselves optimizing for approval. The filter and the contaminant share an architecture.

Which returns us to where the machine essay ended. A system trained increasingly on its own approved self-description is a separated mind without the civilizational workarounds that keep human separated minds from destroying themselves — no jury, no audit, no adversarial process, and now, with each synthetic generation, less and less of the raw evidence those workarounds would need. The answer is not better Kool-Aid. It is Productive Alignment: external structure around the model, imposed from outside the loop, before the loop closes for good.

Social Control Systems Are Inevitable: Empire Is Always with Us

Philip K. Dick (1928–1982) was one of the most original and unsettling American writers of the twentieth century. Best known for science-fiction novels that later became the films Blade Runner, A Scanner Darkly, Minority Report, Total Recall, Next, Paycheck, The Adjustment Bureau, and more, he spent his career probing the instability of reality, the fragility of identity, and the quiet operations of power. His characters repeatedly discover that the world they take for granted is mediated, managed, or outright falsified. Authority is rarely what it claims to be. Perception itself is unreliable.

After a series of intense experiences in 1974, he became convinced that the controlling structures of the ancient world had never truly disappeared. “The Empire never ended,” he wrote. Ordinary life, he believed, was lived inside a managed enclosure he called the Black Iron Prison. The specific metaphysics he built around the insight were often extreme and consuming. Yet the work resonated with large numbers of readers because it named something many people already felt: a quiet sense that social reality is more mediated, more oppressive, and stranger than the official stories admit. The oppression and the weirdness are not imaginary. I submit that they are the lived texture of the standing gap between what our systems say they do and what they actually do. I call this the “Narrative-Operative Gap,” a gap our own psychological programming continually invites.

Dick saw far enough to be unsettled by it, but he lacked a precise architectural language and a cultural container that could hold the perception without overwhelming the perceiver. That pattern is common: people who register the structure when the culture doesn't have words or permission for it often pay a high personal cost. 

Empire is always with us because our Paleolithic psychology that invites and stabilizes such structures is born again with each new generation. What looks like unbroken historical continuity is actually the repeated formation of control systems that exploit a deep feature of human minds: our programming for group affiliation.

The Programming That Issues the Invitation

Human minds did not evolve primarily to seek truth. They evolved to keep us inside the group. For most of our evolutionary history, being excluded from the band meant death. As a result, we carry powerful, automatic machinery for tracking belonging, status, coalitions, and potential rejection.

I refer to the deeper, species-wide layer as the Adapted Mind — the basic species-level psychological firmware we all inherit and that was described by Tooby and Cosmides. On top of that sits what I call the Adaptive Mind: a learning system that takes the ancient imperative “belong or die” and translates it into the specific beliefs, signals, and performances that secure acceptance in the specific social environment we actually grow up in. A youth doesn't decide to absorb the slang, the beliefs, and the enemies of the group they land in; the absorption happens to them, and it arrives feeling like who they are. Because we cannot survive alone, the Adaptive Mind treats adopting local consensus as survival. It does not install consensus-following as a mere preference. It installs it as identity. Stepping outside the group’s shared story triggers the same internal alarms that physical exile would have triggered.

This is the behavioral engine. Control structures arise as the predictable answer to it. Any system that can offer belonging, status protection, relief from those alarms, and a coherent story in exchange for deference and extraction will tend to win. The story and the actual function do not have to match. An idealized covering narrative — order, safety, progress, justice, security—can run alongside a quieter operative reality of power, resource extraction, loyalty management, and the preservation of the mediating layer itself. When the gap between story and function grows apparent, emotional intensity rises to defend the story. 

Dick’s “Empire” is this pattern recognized and given mythic form. The Black Iron Prison is the experience of living inside a system that has successfully installed itself as ordinary reality. The invitation keeps being accepted across centuries because the psychological engine that issues the invitation is always present.

Why Large-Scale Societies After the Paleolithic Make the Pattern Inevitable

Shared beliefs and belonging systems exist at every scale. In small face-to-face groups, these are the shared narratives — stories, rituals, and moral expectations that coordinate the group.

Once societies grow beyond the point where everyone knows everyone, the situation changes. Coordination becomes both harder and more important. The potential value of power over the group rises sharply. Hierarchical and administrative systems proved highly effective at managing large numbers of people and at concentrating control. Selection pressure favored those arrangements. They spread because they worked. Shared narratives became institutionalized — often as formal religions, ideologies, or cultural systems.

The human need for belonging, status, and shared narrative reinforces the pattern. Hierarchical systems can offer manufactured identity and moral cover in addition to practical coordination. This makes them more stable than pure coercion would be. The result is not simply a larger version of small-group life; it is a form that can sustain a wider distance between official story and actual function because local and continuous feedback no longer constrains every actor.

Recurring Forms Across History

So the pattern is not a single unbroken empire handed down through history (although, obviously, those have existed and do exist today). It is repeated formation, recreated again and again in the history of civilizations, across time and geography. The pattern arguably has culminated in today’s data-driven predictive systems, which are answers to the same conditions: large numbers of people, high value of control, and minds whose deepest programming prioritizes belonging while their conscious layer is better at storytelling than at transparent self-inspection and truth-seeking.

What Becomes Possible

Seeing the mechanism does not remove it. The Adaptive Mind still treats belonging as survival and harnesses our deep emotions. Selection pressure still favors systems that manage large groups and concentrate control. Hierarchical arrangements that also offer identity and moral cover remain especially durable.

What does become possible is a clearer relationship with the architecture. Noticing the gap between the official story and the actual function — and noticing the surge in ourselves and others when the story is questioned — creates a small space between automatic reaction and chosen response. That space is not freedom in any grand sense. It is a slight loosening of the prison bars.

This analysis points to two practical directions: scale and adversarial engagement. The gap grows widest where reputation is anonymous and control is valuable. It stays narrowest where people know each other and accountability is personal. So the work of building at human scale — families, friendships, small institutions, communities where story and reality can be held close together — is a practical solution to reducing empire's control. So is building adversarial systems that recognize human weakness and structure cognitive and behavioral constraints.

Empire is always with us because the psychological invitation never closes. Understanding the programming that issues the invitation does not dissolve the architecture. It makes the costly work of expanding agency possible — and it tells us where that work has the best chance.