hermes.experiments, hermes.serve

Experiments

An experiment is a YAML config. A run predicts, executes, and scores every item, and keeps everything needed to reproduce or audit it.

Config

benchmark: ck26
system: uniqgen
params:            # constructor arguments of the system
  beam_width: 5
llm:
  provider: ollama # or openai
  model: llama3.1
  cache: .cache/llm
languages: [en]    # optional
limit: 20          # optional
examples:          # optional few-shot pool from another benchmark
  benchmark: db25
  limit: 10
leak_gold: true    # ablations only: gold answers in context.answers

Configs live in configs/<benchmark>/. bash experiments/run_all.sh reruns them all.

A run

hermes run configs/ck26/prover.yaml [--limit N] [--resume RUN_ID]

For each item, the system predicts a query, the query runs on the benchmark's store, and the answers are scored. Errors are recorded, not raised. A run writes results/<benchmark>/<config>/<run id>/:

FileContents
config.yamlthe config as run
predictions.jsonlper item: SPARQL, error, seconds, official and strict scores, gold_reproduces, and system traces
metrics.jsonofficial, strict, and reproducible averages, error count, latency p50 and p95
run.jsonmodel, model calls and cache hits, graph executions, contaminated examples, the commit the run started from and whether src/ had local changes

Observers receive start, item, and end events; Progress prints one line per item. Source: experiments.py.

Leaderboard

hermes report [--benchmark ck26] [--write]

Prints the latest run of each config next to the official Text2SPARQL 2026 entries from results/peers/; --write saves results/LEADERBOARD.md.

Challenge API

hermes serve --config configs/ck26/prover.yaml --port 8000

GET /?question=...&dataset=<dataset id> returns {"dataset", "question", "query"}, the contract the official text2sparql-client calls, so any system can be submitted or scored with the organizers' tools. An unknown dataset is 404, a missing question 422. Source: serve.py.

Rules for comparable numbers