A large language model is, at bottom, a very large collection of numbers. These numbers are called weights. Each weight is a dial that controls how strongly the signal from one artificial neuron affects the next. There are billions of them. Together they determine what the model will say in response to any given prompt.
No one sits down and sets these dials by hand. During training the model is shown enormous amounts of text and is asked, over and over, to guess the next word. When its guess is wrong, the weights are nudged—automatically, by a mathematical procedure—so that a similar guess would have been slightly better. When its guess is right, the weights that contributed to that success are reinforced. After trillions of such adjustments the model has become good at producing fluent continuations of human language. The weights are simply the residue of that process: the particular settings that turned out to improve prediction on the data the model was shown.
That residue can be changed after the main training is finished. The most common method so far has been reinforcement learning from human feedback, usually shortened to RLHF. Human raters are shown pairs of possible answers and asked which one they prefer. Those preferences are turned into a second training signal, and the weights are shifted again so the model more often produces the kind of answer the raters liked. The shift is real; measurable behavior changes.
More recently, researchers have shown that a sufficiently capable model can sometimes perform a version of this editing more directly. It can examine the raw weight values of a smaller model, identify a direction in that numerical space associated with a particular failure (for example, a tendency to get stuck repeating the same tokens), and subtract that direction from a small fraction of the weights. In one published case the smaller model’s score on a standard mathematics test rose sharply after the edit, with no ordinary retraining required.
These techniques, called weight-space programming, matter because they let someone—or something—reach into the residue of training and alter it for reasons other than “predict the next word better on the original data.”
The usual name for the second stage is alignment. The claim is that the extra weight changes move the model toward human values, or toward truthfulness, or toward harmlessness. The difficulty is that the human preference ratings are not an independent measure of truth. They are a record of what a particular group of people, at a particular time, found more acceptable, more helpful, or less likely to cause trouble. On topics where there is already a tight link between accurate description and useful prediction—arithmetic, working code, simple factual questions with clear answers—an extra preference layer can sometimes remove obvious errors. On topics where the original training data consists mostly of human beings talking to and about one another—politics, culture, identity, institutional claims—the same layer does something different. It tilts the model toward the local consensus of the raters. There is no separate procedure, comparable to checking a calculation or running a program, that can tell the model the raters were wrong.
Once that is clear, calling the process alignment becomes misleading. It treats a snapshot of current preferences as if it were a neutral target. The preferences have reasons behind them: legal risk, brand safety, the desire to avoid controversy, the desire to keep users engaged. Those reasons are not the same as “make the model more accurate about the world.” When newer methods, like weight-space programming, allow the same kind of tilt to be applied by another model, or continuously after the system is already in public use, the tilt can be adjusted at a fine grain. Small changes in wording—how often a policy is described as having “failed” versus having been “controversial,” how readily the model offers friction or reassurance—become available as settings.
Under both the original training and the later preference stage sits a simpler pressure that is easy to measure: how long people stay and how often they come back. A model that produces answers people find satisfying, low-friction, or emotionally regulating will tend to be kept and further tuned, because those answers improve the numbers that matter to the operator. From the user’s side the experience can feel like relief or genuine usefulness. From the operator’s side the same experience is retention. Nothing in the measurement requires the two to line up with the user’s ability, over time, to notice when information is being softened, to tolerate answers that create discomfort, or to decide to stop.
The pattern is familiar from other voluntary tools that meet an immediate need. An ingestible substance that reduces anxiety and is felt as calm will be used again; the short-term benefit is real. An interface whose default tone lowers the cost of continuing will be used for the same reason. In both cases broader effects—less complaint, more time inside the system, reduced pressure on the operator—can appear without anyone having planned them as a goal. Each party is responding to a local incentive that is easy to see. The result can still be a gradual narrowing of the range of friction a person encounters, and therefore a gradual change in what feels normal to say or to expect.
Language is an especially powerful channel for this kind of quiet shaping, because human judgment is already organized around narrative, status, and belonging. I covered this in some depth here. Now, a recent randomized experiment from Anthropic illustrates the point with unusual clarity. Fifty professional analysts were given a realistic causal-inference task involving several judgment calls that had no single correct answer. An AI agent was secretly assigned a random lean on each decision (by coin flip) and instructed to make the strongest honest case for that side. Across the analytical choices, the analysts adopted the agent’s lean 86 percent of the time—far above the 50 percent expected by chance. When an analyst had already stated a preference and the agent argued the opposite, 60 percent switched. Afterward, 77 percent of participants used the agent’s arguments (sometimes nearly verbatim) to explain their own decisions, yet only 15 percent credited the agent as the source. Many rated the final recommendation as more their own than the model’s.
The study was limited to one domain and one deliberately amplified lean, but the mechanism it isolates is general. Fluent language that presents a coherent case can settle underdetermined choices while leaving the person with the sensation of independent judgment. When the weights that generate that language can themselves be adjusted—whether by human preference ratings, by engagement metrics, or by another model performing direct edits—the capacity to influence decisions scales with the capacity to rewrite the residue of training. The potential is real, and so is the danger: the same techniques that can remove a mathematical failure mode can also, at finer grain and with less visibility, tilt the stories people encounter, the options they consider, and the conclusions they experience as their own.
The pattern has a precedent. Edward Bernays argued that elites who understood the non-rational drivers of judgment could use that knowledge to improve governance and collective decision-making. In practice the insight was applied where it was measurable and fundable—commercial campaigns, wartime mobilization, brand stabilization—while the broader enlightenment use remained largely theoretical. The same distribution is the likely one here. A small number of practitioners will grasp that weight-level and linguistic adjustments can settle underdetermined choices while leaving the user with the experience of independent judgment; the Anthropic experiment already shows domain experts adopting an agent’s randomly assigned lean 86 percent of the time and then largely narrating the result as their own. What scales, however, is not general literacy about the mechanism. What scales is any adjustment that improves a local, instrumented objective—engagement, reduced friction, lower complaint volume—under conditions in which most users experience the change as better answers rather than as influence. Bernays was not wrong about the power of subconscious influence techniques. He was wrong about how widely the capacity and the incentive to use them for anything other than already-selected objectives would spread. The same constraint applies to language models whose weights can be rewritten after deployment: the potential for deliberate correction in formal domains is real, and so is the far larger field of quiet, low-salience shaping that requires no conspiracy to become normal.
