hermes.benchmarks

Benchmarks

Every benchmark, whatever its source format, is stored the same way, so every system is scored the same way.

The suite

Name	Questions	Items	Graph	License
`ck26`	50	100 (en, de)	Text2SPARQL corporate graph, local	CC BY-SA 4.0
`db26`	50	100 (en, es)	DBpedia, organizers' endpoint	CC BY-SA 4.0
`ck25`	50	50 (en)	corporate graph, local	CC BY-SA 4.0
`db25`	100	200 (en, es)	DBpedia, organizers' endpoint	CC BY-SA 4.0
`qald10`	394	1,564	Wikidata, live	MIT
`qald9plus`	150	450	DBpedia, live	CC BY 4.0

Format

data/graphs.yaml                       local files or an endpoint per graph
data/benchmarks/<name>/benchmark.yaml   graph, languages, license, source URLs and SHA-256
data/benchmarks/<name>/questions.jsonl  one row per question and language
data/benchmarks/<name>/gold.json        gold answer sets

{"id": "ck26:7-en", "qid": "7", "lang": "en", "question": "...", "sparql": "...",
 "features": ["SELECT", "ORDER"], "order_matters": false}

Ids follow the official scorer. Gold answers use its layout: every value of every variable, flattened into one set. Sources are pinned to commits. Adapters convert Text2SPARQL YAML and QALD JSON; a new benchmark is an adapter and an entry in sources.py. Source: benchmarks/.

Commands

hermes data list                 # known and downloaded benchmarks
hermes data fetch ck26 db26      # download, normalize, fetch local graphs
hermes data verify ck26          # run every gold query and compare with gold

Caveats

Read first

These facts change how any number on these benchmarks should be read.

CK26 is CK25 paraphrased. 49 of the 50 CK26 gold queries appear verbatim in CK25, on the same graph. Neither is a development set for the other. Runs count overlapping few-shot examples as contaminated_examples.
The CK gold answers depend on the engine. They were computed on Tentris. On Oxigraph, 43 of 50 CK26 gold queries reproduce; the rest differ in tie-breaking under LIMIT, number formatting, or the non-standard xsd:int(). All 50 DB26 gold queries reproduce on the organizers' endpoint.
QALD answers drift. QALD-9-plus targets DBpedia 2016-10 and QALD-10 was answered on Wikidata in 2022; the live endpoints have changed.