Embedded Systems Series · Paper 4 of 5

The information environment

Synthetic Information and Strategic Influence

Cheap generation has not made propaganda much more persuasive. It has made faking cheap relative to checking. Past a point, the rational response to that is to stop believing what cannot be checked, and true information pays the price.

Adapted from Working Paper P18September 2026About 18 minutes to read

In brief

1. AI-written propaganda roughly matches human-written propaganda, gains little from bigger models or static microtargeting, and has shown null effects on attitudes and votes. Evidence
2. What AI changed is the cost and volume of production. Verification is still slow and sparse. Evidence
3. In a simple model, abundant unverifiable content tips audiences from default trust to default scepticism: false beliefs fall while true information stops landing. Model
4. Personalisation matters mainly because it defeats collective fact-checking. The contest shifts toward authentication. Inference

How claims are tagged: Evidence Model Inference Hypothesis Framework

The danger that was expected, and the one that arrived

When generative models became widely available, the expected danger was deception at scale: fake video indistinguishable from real footage, and machine-written propaganda more persuasive than anything a troll farm could produce. The first expectation has largely been met technically. Synthetic faces are rated not only indistinguishable from real ones but more trustworthy, listeners identify synthetic speech about three times in four, and readers cannot tell AI-written self-descriptions from human ones.¹ The second has not been met in the form expected.

Figure 1

AI-written propaganda persuades about as well as the real thing

Share of 8,221 US adults agreeing with the thesis of real covert propaganda articles. Raw GPT-3 output came close to the original. Discarding outputs that missed the thesis made it statistically indistinguishable, and a light editing step with a best-of-three choice exceeded it. Persuasiveness also shows sharply diminishing returns to model size.²

Goldstein et al. (2024), PNAS Nexus; survey experiment of December 2021. Evidence

What changed is volume. DC Weekly, a fabricated US news site identified as part of a Russian influence operation, raised its daily output 2.4-fold after it began rewriting material with a language model, widened its topics and kept its persuasiveness.³ Meanwhile the covert operations OpenAI disrupted in 2024 reportedly had no significant effect on audience engagement.⁴ AI has transformed the economics of producing influence content far more than the potency of any single piece.

That would be reassuring if production were the only economics that mattered. It is not. Every piece of content a person or institution acts on must be either verified or trusted, and verification remains slow, costly and scarce. When producing plausible content becomes nearly free and checking it does not, a rational audience first believes more falsehoods and then, once unverifiable content is abundant enough, believes less of everything it cannot check. Past that point, scepticism protects people from deception and strips true information of its reach.

The chain, and where AI enters

Influence travels a chain from generation to strategic adaptation. AI enters directly at two links, generation and personalisation. Everything downstream is a response, and the responses decide the consequences.

Figure 2

The influence chain and the state of the evidence at each link

GenerationVolume up 2.4× in one field case; potency per item flatEstablished

PersonalisationLive dialogue persuades; static microtargeting adds littleSplit

Targeting and exposure1% of users saw 70% of one campaignEstablished

DistributionFalsehood's speed advantage is contestedContested

InterpretationHuman detection unreliable; warnings do not helpEstablished

BehaviourEffects on attitudes and votes small or nullEstablished

TrustUncertainty lowers trust; interventions discount true contentEstablished

VerificationUnder half of narratives checked, median four days lateOne election

CountermeasuresModest effects, and they scale with AI tooEstablished

Strategic adaptationShift to volume and laundering in one case; outsourcing forecastOne case

Shaded boxes are where AI enters directly. Adaptation feeds back to generation: attackers shift toward volume, defenders toward authentication and machine-speed correction.

What the evidence shows

The evidence on persuasion is genuinely split, and the split is informative. In live debate, GPT-4 given basic personal information about its opponent out-persuaded humans in 64.4% of pairs where the sides differed; an author correction to that study has since been published.⁵ Yet among 8,587 people, static messages microtargeted by GPT-4 were no more persuasive than generic ones (effects of 4.83 against 6.20 percentage points).⁶ And 59 experiments with 34,000 people found small average effects of political advertising on candidate choice.⁷ Evidence

The single most important finding for this argument is a symmetry.

Figure 3

The same models argue as well for a falsehood as against it

Change in belief in a conspiracy theory, in points, among 3,996 Americans after a dialogue with a language model instructed to argue against it or for it. To a first approximation, persuasive capacity is indifferent to truth. One asymmetry favours truth: debunking changed the social-media posts participants wrote; arguing for the conspiracy did little.

Costello et al. (2026), preprint.⁸ Evidence

Taken together: the output one operation could produce rose sharply once it adopted generative tools; the potency of each item has not risen much; and the costs that are measurable include lost trust and lost credibility of true information.⁹ Evidence Inference on production cost

A model of default trust

Consider someone who, on some topic and over some period, meets G genuine messages and S fabricated ones, and can verify at most V of them. The limit is set by time, attention and tools, not willingness. In this model "synthetic" means false: true content made by a machine counts as genuine. Acting on a true message is worth g; acting on a false one costs b. Unverified messages must be accepted or rejected on the base rate alone, and accepting them is worth it only while the share of genuine messages stays above a threshold.

Accept unverified content while θ = G / (G + S) ≥ θ* = b / (g + b)  ·  Tipping volume S* = G × g / b

Below the threshold, the rational default is to reject what cannot be checked. Explore what happens as synthetic volume rises. Model

Figure 4

Default trust, and how it collapses

Fabricated messages, per 100 genuine

75

Logarithmic slider, from 10 to 10,000.

Messages the audience can check, V

50

Cost of acting on a falsehood, relative to the value of acting on the truth

Equal 2× 4× (a crisis) 9×

Default trustthe audience's rational default

53.6false messages believed

0genuine messages lost

100tipping volume S*, per 100 genuine

False messages believedGenuine messages lostTipping point

Before the tipping point, false beliefs rise with synthetic volume. After it, people reject what they cannot check: false beliefs drop to zero while genuine messages go unbelieved. With 100 genuine messages, 50 checks and equal costs, false beliefs reach about 75 just before parity; genuine losses jump to about 75 just after it, and reach 95 when fabrication is ten times genuine volume.

Stated model with illustrative parameters; it establishes a structure, not a magnitude.¹⁰

The measurement trap

Belief in falsehood, a quantity often measured on its own, improves across the switch, while the quantity that matters for a working information environment, how much true information reaches the people it concerns, deteriorates. An evaluation that tracks only false belief would record success at the moment the environment fails. Measures of discernment, which also count belief in true content, would register the loss. Model

The trust findings above are the fingerprints this structure predicts: uncertainty rather than deception after deepfakes, interventions that lower the credibility of true information, labels that discount true content. None shows that any real population has crossed a tipping point, and this article does not claim one has. They show that the mechanism the second regime depends on operates and can be measured. Inference

Why crises are the dangerous moment

With equal costs, an adversary must match honest volume to tip an audience. If acting on a falsehood costs four times what acting on the truth is worth, as with a fake evacuation order or a spoofed payment instruction, the threshold share rises to 0.8 and a quarter of genuine volume is enough; at nine times, a ninth. In a crisis the value of believing true information also rises, which pushes the other way. The tipping volume still falls if the cost of error rises faster, or if genuine information becomes scarce. On the day of the Iberian blackout in April 2025, Spain's prime minister warned of perfect conditions for misinformation. No adversary was needed to fill that vacuum.¹¹

Personalisation defeats the amortisation of checking

Most verification that matters is collective: a fact-check, a platform label or a news correction serves everyone who meets the same claim, and that is what makes verification affordable. With F checks available, I impressions of synthetic content and n impressions per distinct variant, at most F × n / I of impressions can be covered, and the cost of verification per impression is the cost of a check divided by n. Broadcast propaganda, one claim seen by millions, is cheap to check per person reached. Content generated afresh for each person makes every impression its own variant.

Figure 5

Fifty fact-checks against a million impressions

Impressions per distinct variant

10,000

100,000 is broadcast propaganda; 1 is content made for each person.

50%share of impressions any check can cover

×10verification cost per impression, against broadcast

Share of impressions covered

Verification cost per impression (cost of one check = 1)

Fifty checks cover every impression when each variant is seen 100,000 times, half when it is seen 10,000 times, and one impression in 20,000 when every variant is unique. Generation cost per impression falls as models get cheaper to run, while verification cost per impression rises as variants multiply, so the gap moves as the product of both trends.

Stated model with illustrative numbers. Model

So the strategic significance of personalisation lies less in persuasion, where the evidence is split and effects small, than in verification, where the effect is structural. The same logic applies to AI assistants that retrieve and summarise for their users: where evidence is aggregated as if its pieces were independent, an adversary who controls how often an assertion appears, rather than how persuasive it is, can control the result.¹² Inference

Verification capacity is the lever, and authentication the chokepoint

For a given volume of fabrication, the one parameter in the model that helps in both regimes is V, the capacity to verify: it lowers false belief before the tipping point and restores the reach of true information after it. Authentication, signing genuine content at its source, can in principle cut the cost of checking a claim to a constant number of actions. One preprint's laboratory study reported verification time falling 73%, from 15.7 to 4.2 seconds, with signed evidence bundles and no loss of accuracy.¹³

Figure 6

Authentication splits the environment in two

Signed channelGenuine content signed at the source; checking a claim is nearly free

Unsigned channelEverything else; checking is slow and costly

Default trust can surviveUnsigned content is excluded, so volume of fabrication does not dilute it

Tends to default scepticismAs fabrication grows, less of anything unverifiable is believed

Two limitsSigning authenticates origin, not truth. And the first independent formal analysis of the main content-provenance specification found it does not meet its own security goals.¹⁴

Whoever controls signing keys, provenance standards and verified channels acquires the strategic position broadcast towers once held. A state that controls authentication can decide whose content is presumed genuine. The question of who signs is a question of power, not only engineering. Inference

Strategic consequences

Attackers shift from persuasion to saturation. If persuasion per item has saturated and is indifferent to truth, and the damage runs through base rates and verification capacity, the efficient strategy is to raise volume, multiply variants, strike when verification is slowest, and target the channels genuine information depends on. That fits the move to volume observed in one operation and the forecast of propaganda sold as a service. It also implies that the most dangerous moment is a crisis rather than an election campaign. Inference

Attribution gets harder. States accused of cyber operations favour "non-denial denials" that keep their role ambiguous.¹⁵ Carelessness, such as posting a model's refusal message verbatim, has revealed AI use in some operations, but the operations themselves were exposed by a platform's own monitoring and by journalists working with researchers.

The state's own countermeasure can destroy verification. Shutting down channels lowers genuine information and verification capacity together, which on the model moves an audience toward the regime where nothing unverifiable is believed, at the moment true information matters most.

Figure 7

How a shutdown can raise the harm it targets

Connectivity shut downto stop misinformation or coordination

Genuine information fallsG drops

More unverified claims, and the incidents they driveby other routes: word of mouth, cached content, SMS

Checking capacity fallsV drops

The companion paper on nth-order reasoning works through a closely related case, a connectivity restriction imposed to suppress disorder, as a stated model, and finds this route among the three that carry its argument. A qualitative study argues that Sudan's 2019 shutdown produced conditions favourable to state disinformation.¹⁶ Model

Objections, briefly

"The fears are overblown."

On persuasion, the evidence agrees: exposure is concentrated and measured effects on votes are small or null. If the damage runs through base rates and verification capacity, though, small persuasion effects are compatible with large costs, and reassurance drawn from null effects on votes is reassurance about the wrong quantity.

"People are sceptical, not credulous."

Some evidence shows audiences discounting what they see, particularly after deepfakes, corrections or labels. The model predicts exactly that. What it adds is the cost of scepticism: the reach of true information.

"Detection and provenance will solve it."

Perhaps for signed channels. Detectors fail on unfamiliar generators, provenance can certify only what is signed, and partial labelling raises the perceived authenticity of what is not.

"Conversational AI with memory will prove far more persuasive."

The most serious objection, and it may prove right. The null result for static microtargeting need not carry over to persistent assistants. If it does not, persuasion will matter more than this article allows, and verification will be worse, because each conversation is its own variant.

What to do

What this analysis cannot tell you

The model treats verification as accurate, audiences as identical, the period as static and the share of genuine content as known. Real audiences act on a perceived share, which coverage of misinformation can lower with no change in the actual share, and a real tipping point would be spread out rather than sharp. Several findings rest on single studies, preprints, abstracts or platform self-reports, and the most prominent evidence on conversational persuasion has a correction whose content could not be checked. The evidence is mostly American and European; Indian evidence comes from public WhatsApp groups that may not represent private chats. The first effect of near-free generation is more content, not more persuasion. The consequential effect falls on the base rate of genuine content, on the collective checking that personalisation defeats, and on authentication, which becomes the point of control.

Notes

1. Nightingale and Farid (2022) on faces; Mai et al. (2023) on speech (73%, n = 529); Jakesch, Hancock and Naaman (2023) on text. Abstracts. ↩
2. Goldstein et al. (2024), full text; Hackenburg et al. (2025), PNAS, on diminishing returns to model size. ↩
3. Wack et al. (2025), PNAS Nexus: one outlet, a before-and-after design. ↩
4. Claburn (2024), The Register, reporting OpenAI's May 2024 report, a platform self-report. ↩
5. Salvi et al. (2025), Nature Human Behaviour, N = 900: an 81.2% increase in the odds of higher agreement. An author correction was published on 3 September 2026; its content could not be retrieved, and the figures are those originally published. ↩
6. Hackenburg and Margetts (2024), PNAS. ↩
7. Coppock, Hill and Vavreck (2020), Science Advances. ↩
8. Costello et al. (2026), arXiv:2601.05050: −12.1 points when arguing against, +13.6 when arguing for; the sharing result comes from a study of 1,272. ↩
9. Eady et al. (2023); Grinberg et al. (2019); Köbis, Doležalová and Soraperra (2021); Groh et al. (2022); Vaccari and Chadwick (2020); Hoes et al. (2024), with an author correction; Altay and Gilardi (2024); Peloquin-Skulski et al. (2025), preprint; Wack, Duskin and Hodel (2024), preprint; Walter and Murphy (2018). ↩
10. G = 100 genuine messages; the crisis case sets b = 4g. Values at the switch: false beliefs 74.1 at 99 fabricated messages; genuine losses 75.1 at 101. ↩
11. Euronews (29 April 2025), as recorded in the programme's research log. ↩
12. The programme's survey of memory-augmented language models, The Currency of a Fact, Proposition 20.2, derived there and untested. ↩
13. Luberisse (2025), a single-author preprint that has not been peer-reviewed; 240 laboratory participants. ↩
14. Golaszewski et al. (2026), preprint, on the C2PA specifications; Nemecek et al. (2026), preprint, on contradictions between provenance manifests and watermarks; Le et al. (2023) on detectors. ↩
15. Brown and Fazal (2021), abstract. The programme's six-layer channel model places synthetic media at its fourth layer, integrity, and information operations at its fifth, interpretation. ↩
16. The programme's working paper Nth-Order Thinking for Policy and Government Decisions, its stated connectivity-restriction model; Bhatia et al. (2023), a qualitative single case. ↩
17. Liu, Rahimian and Garimella (2025), citing the user figure (a secondary figure); Agarwal, Shahid and Vashistha (2024); Reis et al. (2020). ↩
18. Guess et al. (2020), PNAS, corrected in 2023; Liu, Rahimian and Garimella (2025), citing Arun (2019), and Medeiros and Singh (2020) on the lynchings, for which no official count was found; Malik (2025) on shutdown orders; Access Now, #KeepItOn. ↩

Adapted for general readers from Working Paper P18 of the Embedded Systems Series (September 2026). The working paper carries the full argument, the provenance of every figure and the complete reference list. Model results are computed from a stated model and are not estimates of any real information environment.