Results

Leaderboard

Every Hermes run next to the official systems of the Text2SPARQL 2026 challenge, ranked by the official scorer.

Hermes is a research system, and it does not lead this board: the best official system scores well above every Hermes run on both tracks. The board shows that on purpose. It is useful for what it separates inside Hermes: the planner with a verifier (prover) beats the unverified algebra agent, which beats one-shot generation, and larger Qwen3 models help.

How to read it

• Set F-nDCG is the challenge's headline metric: set F1 over the returned rows, combined with ranking quality.
• Hermes rows are runs of this repository (system and model in the name, @qwen3-14b for example). Official rows are the published scores of the challenge's other systems, shown for scale.
• LLM calls per item and GPU minutes are what a run cost; official systems report neither.
• Click a column to sort. Ranks stay with the set F-nDCG, so a sort does not change who is first.

The numbers come from results/LEADERBOARD.md, which hermes report --write generates from every run's metrics.json; see Experiments.