hermes.evaluation
Evaluation
The scorer reproduces all 5,024 published Text2SPARQL 2026 scores, so Hermes numbers sit on the leaderboard's scale.
Metrics
The official scorer flattens every value of every variable into one set and scores it with trec_eval: set precision, set recall, and set F1 per question, nDCG where order matters, and the ranking metric set_F_ndcg, which uses nDCG on those questions and F1 elsewhere. score(result, gold, order_matters, strict) scores one question; aggregate(per_item, languages, strict) gives overall and per-language averages. Source: evaluation.py.
Parity
experiments/check_scorer.py downloads every team's predictions and official scores for CK26 and DB26 from the pinned challenge repository, rescores them, and compares all 5,024 per-question and average numbers. None differ.
Official quirks and the strict variant
| Official behavior | Strict |
|---|---|
An ASK answer is a document named true. A gold true scores 1.0 whatever the answer; a gold false scores 0.0 for everyone. | Booleans score by exact match. |
| nDCG ignores the returned order, since every value has the same score. | nDCG uses the returned order. |
set_F_ndcg counts the scope's mean nDCG as one extra item. | A plain mean. |
Runs report both, side by side.
Reproducible subset
Each run also reruns every gold query on its own store and flags gold_reproduces. The reproducible averages restrict the official metrics to those items, which separates a method's errors from differences between graph engines.