Experiments

Leaderboard

Attack strategies and defenses ranked on four real ontologies, from the recorded experiments.

Two boards, from the same experiment. In the first the strongest attacker ranks first: the search that finds the most harmful chain of edits at equal cost. In the second the best defense ranks first: the one that leaves the least harm. Aegis's own searches and defenses are highlighted next to the simple baselines they have to beat.

Attack: who finds the worst chain?

Defense: who leaves the least harm?

How to read it

• Harm is the mean F1 drop over the five "What kinds of X are there?" queries, from 0 to 1.
• Attack compares the searches at equal evaluations: beam search runs to completion and sets the budget that random search and UCT get. Beam can win at small budgets; Attack search shows how UCT catches up as the budget grows.
• Defense is against a two-edit attacker. Residual harm is the best of beam search and UCT against the locked ontology, so a lock set is not credited for a chain the search missed.
• Click a column to sort. Ranks stay with the headline column.