Verified query planning over the SPARQL algebra

Hermes answers a question over a knowledge graph by planning a SPARQL query one algebra operator at a time. A verifier types every step against the RDF schema before it runs, and execution on the graph tells the planner what each step did.

$ git clone https://github.com/lamng3/hermes.git && cd hermes
$ pip install -e ".[dev]"
$ hermes generate --graph corporate --system prover --question "..."

The algebra Playground Benchmarks

Four systems, one question

recorded benchmark runs

Try it: Pick a question, then switch systems to compare the SPARQL each one wrote and how it scored.

Question

Leaderboard

set F-nDCG, ck26

Qwen3-14B on 100 questions of the Text2SPARQL 2026 corporate benchmark; answers are scored by the benchmark's official scorer. Several official systems score higher than every Hermes run, and the leaderboard shows them. The page computes nothing: it shows hermes/results/ck26.

This site documents the code of the paper Hermes: Verified Query Planning over the SPARQL Algebra with Language Agents (Lam Nguyen, Case Western Reserve University). The repository is its official implementation.

The idea

Language models usually write a whole SPARQL query, then repair it after it fails. A query optimizer works differently: it applies one algebraic operator at a time and estimates its effect before keeping it. Hermes takes the optimizer's view of query generation:

A typed SPARQL algebra. A query plan is a term of the algebra. The RDF schema gives every variable a type, and rules decide without running the plan whether it is valid. See Verifier.
Verified query planning. Actions are algebra operators. After each one the planner sees the number of solutions, how it changed, a sample of rows, and any error. A plan is accepted only when it is valid and meets a goal stated from the question. See Algebra.
One interface for methods. One-shot generation, constraint-guided search (UniQGen), and the planners are swappable systems on the same graphs and the same scorer. See Systems.
A reproducible evaluation. Six benchmarks in one format, and a scorer that reproduces every published Text2SPARQL 2026 score. See Benchmarks and Evaluation.

The pieces

AlgebraPlans, actions, the explorer, and the feedback after every step. VerifierTyping, well-formedness, goals, and the prover system. SystemsThe registry and the four shipped systems. Graph storesOxigraph, rdflib, and SPARQL endpoints behind one query call. BenchmarksText2SPARQL and QALD in one normalized format, with caveats. EvaluationOfficial, strict, and reproducible-subset metrics. ExperimentsConfigs, runs, the leaderboard, and the challenge API. ModelsOllama or an OpenAI-compatible API, with retries and a replay cache.

Quick start

hermes data fetch ck26                      # download and normalize a benchmark
hermes run configs/ck26/algebra-agent.yaml  # predict, execute, score
hermes report                               # leaderboard with official peer scores

One question at a time:

hermes generate --graph corporate --system prover \
  --question "Who is Heinrich Hoch's manager?"

from hermes import KnowledgeGraph, ask

kg = KnowledgeGraph.from_files("onto.ttl", "data.ttl")
result = ask("Which cities hosted the Winter Olympics?", kg, system="prover")
print(result["sparql"])

Install

git clone https://github.com/lamng3/hermes.git
cd hermes
pip install -e ".[dev]"
pytest

The install name is hermes-sparql and the import name is hermes. Python 3.11 or newer. Dependencies are rdflib, pyoxigraph, httpx, and pyyaml. The tests need no model and no network.