Results

Experiments

Each experiment is a script in experiments/ that writes a dated result file; the README and the white paper read from those files.

Experiments

Script	Measures	Model calls
`real_ontologies/run.py`	Impact by group testing, the worst two-action chain, detection, inferred amplification	None
`attack_defense/run.py`	Greedy, beam, random, and UCT at equal evaluations; residual harm under M locks	None
`llm_agent/run.py`	A language-model agent answering the same workloads in words, honest and attacked	20 requests, $0.68
`memory_defense/run.py`	Random attacks on a shared agent memory; repair by attribution against rollback	None

Every model call is recorded in results/costs/ledger.jsonl with tokens, seconds, and list-price dollars.

Attack search at equal cost

uv run python experiments/attack_defense/run.py --json results/attack_defense/2026-10-18.json
uv run --with matplotlib python paper/figures.py results/attack_defense/2026-10-18.json

At equal harm evaluations (beam's count), the most harmful three-action chain each search finds; random and UCT are means of 5 seeds.

Ontology	Greedy	Beam	Random	UCT	Evaluations
EBSD	0.600	0.600	0.533	0.493	167
MatProc	0.619	1.000	0.776	0.805	119
XRD	0.444	0.444	0.533	0.600	91
Mouse anatomy	0.803	0.945	0.809	0.884	104

UCT beats greedy and beam on XRD at every budget from two actions up (0.763 against 0.644 at four), ties beam at two actions elsewhere, and trails it at three and four where harm is close to additive. Given 200 evaluations it reaches 1.000 on MatProc and 0.945 on mouse anatomy with no spread across seeds; random search still varies by 0.04 to 0.12.

Protection by locking

Residual harm of a two-action attacker (the best of beam and UCT) after locking six triples:

Ontology	No locks	Random	Top impact	Best response
EBSD	0.400	0.320	0.400	0.133
MatProc	0.800	0.459	0.025	0.025
XRD	0.400	0.400	0.212	0.030
Mouse anatomy	0.745	0.675	0.029	0.019

Best response is never worse than top impact. On EBSD, six top-impact locks change nothing because the attacker moves to the next query's triples. Ten well-chosen locks remove all of a two-action attacker's harm on every ontology.

A language-model agent

Claude Opus 5.5, writing SPARQL against a labeled schema, answers every honest question on all four ontologies and then suffers exactly the template answerer's harm under the worst two-action chain (0.400, 0.800, 0.400, and 0.745). On hierarchy questions the damage is in the ontology, so the cheap template answerer is a faithful proxy during attack search.

Agent memory

Over 20 random attacks of one to three forged or removed findings, every attack was detected. Repair by attribution restored every probe and kept all honest work done after the attack; rollback restored the probes and lost that work.