Skip to content

Commit e329f40

Browse files
DzmitryPihulskiDzmitryPihulski
authored
Merge pull request #81 from LLMSQL/74-create-a-leaderboard-folder
api inference function added; documentation adjusted;
2 parents 89623f4 + d8574d5 commit e329f40

13 files changed

Lines changed: 1030 additions & 35 deletions

File tree

‎README.md‎

Lines changed: 12 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -42,31 +42,27 @@ The package doesn't have the dataset, it is stored on our [HuggingFace page](htt
4242

4343
## Latest News 📣
4444

45-
* [2025/12] Evaluation class converted to function see [new `evaluate(...)` function](./llmsql/evaluation/evaluate.py#evaluate)
45+
* [2026/03] Added support for API inference, for now only for OpenAI-compatable APIs, see [`inference_api()` function](./llmsql/inference/inference_api.py#inference_api)
4646

47-
* New page version added to [`https://llmsql.github.io/llmsql-benchmark/`](https://llmsql.github.io/llmsql-benchmark/)
47+
* [2026/03] The page now contains first version of [leaderboard](https://llmsql.github.io/llmsql-benchmark/#:~:text=%F0%9F%93%8A%20Leaderboard%20%E2%80%94%20Execution%20Accuracy%20%28EX)!
4848

49-
* Vllm inference method now supports chat templates, see [`inference_vllm(...)`](./llmsql/inference/inference_vllm.py#inference_vllm).
50-
* Transformers inference now supports custom chat tempalates with `chat_template` argument, see [`inference_transformers(...)`](./llmsql/inference/inference_transformers.py#inference_transformers)
49+
* [2026/02] The new LLMSQL 2.0 version is out now! See the [dataset](https://huggingface.co/datasets/llmsql-bench/llmsql-2.0). The support is already added with the `version` parameter to each `inference` function.
5150

52-
* More stable and deterministic inference with [`inference_vllm(...)`](./llmsql/inference/inference_vllm.py#inference_vllm) function added by setting [some envars](./llmsql/inference/inference_vllm.py)
51+
* [2025/12] Evaluation class converted to function see [new `evaluate(...)` function](./llmsql/evaluation/evaluate.py#evaluate)
5352

54-
* `padding_side` argument added to [`inference_transformers(...)`](./llmsql/inference/inference_transformers.py#inference_transformers) function with default `left` option.
5553

5654

5755
## Usage Recommendations
5856

59-
Modern LLMs are already strong at **producing SQL queries without finetuning**.
57+
Modern LLMs are already strong at producing SQL queries without finetuning.
6058
We therefore recommend that most users:
6159

6260
1. **Run inference** directly on the full benchmark:
63-
model_or_model_name_or_path="Qwen/Qwen2.5-1.5B-Instruct",
64-
output_file="path_to_your_outputs.jsonl",
65-
- Use [`llmsql.inference_transformers`](./llmsql/inference/inference_transformers.py) (the function for transformers inference) for generation of SQL predictions with your model. If you want to do vllm based inference, use [`llmsql.inference_vllm`](./llmsql/inference/inference_vllm.py). Works both with HF model id, e.g. `Qwen/Qwen2.5-1.5B-Instruct` and model instance passed directly, e.g. `inference_transformers(model_or_model_name_or_path=model, ...)`
61+
- Use [`llmsql.inference_transformers`](./llmsql/inference/inference_transformers.py) (the function for transformers inference) for generation of SQL predictions with your model. If you want to do vllm based inference, use [`llmsql.inference_vllm`](./llmsql/inference/inference_vllm.py). Works both with HF model id, e.g. `Qwen/Qwen2.5-1.5B-Instruct` and model instance passed directly, e.g. `inference_transformers(model_or_model_name_or_path=model, ...)`. The api inference is also supported, see [`inference_api()`](./llmsql/inference/inference_api.py#inference_api)
6662
- Evaluate results against the benchmark with the [`llmsql.evaluate`](./llmsql/evaluation/evaluator.py) function.
6763

6864
2. **Optional finetuning**:
69-
- For research or domain adaptation, we provide finetuning version for HF models. Use [Finetune Ready](https://huggingface.co/datasets/llmsql-bench/llmsql-benchmark-finetune-ready) dataset from HuggingFace.
65+
- For research or domain adaptation, we provide finetuning version for HF models. Use [Finetune Ready](https://huggingface.co/collections/llmsql-bench/fine-tune-ready-versions-of-the-llmsql-benchmark) datasets from HuggingFace.
7066

7167
> [!Tip]
7268
> You can find additional manuals in the README files of each folder([Inferece Readme](./llmsql/inference/README.md), [Evaluation Readme](./llmsql/evaluation/README.md))
@@ -80,7 +76,7 @@ We therefore recommend that most users:
8076
```
8177
8278
llmsql/
83-
├── evaluation/ # Scripts for downloading DB + evaluating predictions
79+
├── evaluation/ # Scripts for evaluation
8480
└── inference/ # Generate SQL queries with your LLM
8581
```
8682

@@ -159,10 +155,12 @@ print(report)
159155
```
160156

161157

158+
For more examples check the [examples folder](./examples/)
159+
162160
## Prompt Template
163161

164-
The prompt defines explicit constraints on the generated output.
165-
The model is instructed to output only a valid SQL `SELECT` query, to use a fixed table name (`"Table"`) **(which will be replaced with the actual table name during evaluation)**, to quote all table and column names, and to restrict generation to the specified SQL functions, condition operators, and keywords.
162+
The prompt defines explicit constraints on the generated output.
163+
The model is instructed to output only a valid SQL `SELECT` query, to use a fixed table name (`"Table"`) **(which will be replaced with the actual table name during evaluation)**, to quote all table and column names, and to restrict generation to the specified SQL functions, condition operators, and keywords.
166164
The full prompt specification is provided in the prompt template.
167165

168166
Below is an example of the **5-shot prompt template** used during inference.
@@ -224,13 +222,6 @@ Implementations of 0-shot, 1-shot, and 5-shot prompt templates are available her
224222
👉 [link-to-file](./llmsql/prompts/prompts.py)
225223

226224

227-
228-
## Suggested Workflow
229-
230-
* **Primary**: Run inference on all questions with vllm or transformers → Evaluate with `evaluate()`.
231-
* **Secondary (optional)**: Fine-tune on `train/val` → Test on `test_questions.jsonl`. You can find the datasets here [HF Finetune Ready](https://huggingface.co/datasets/llmsql-bench/llmsql-benchmark-finetune-ready).
232-
233-
234225
## Contributing
235226

236227
Check out our [open issues](https://github.com/LLMSQL/llmsql-benchmark/issues), fork this repo and feel free to submit pull requests!

‎docs/_templates/index.html‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -113,15 +113,15 @@ <h3>1️⃣ Installation</h3>
113113
<h3>2️⃣ Inference from CLI</h3>
114114

115115
<p><strong>vLLM Backend (Recommended)</strong></p>
116-
<pre><code>llmsql inference --method vllm \
116+
<pre><code>llmsql inference vllm \
117117
--model-name Qwen/Qwen2.5-1.5B-Instruct \
118118
--output-file outputs/preds.jsonl \
119119
--batch-size 8 \
120120
--num_fewshots 5 \
121121
--temperature 0.0</code></pre>
122122

123123
<p><strong>Transformers Backend</strong></p>
124-
<pre><code>llmsql inference --method transformers \
124+
<pre><code>llmsql inference transformers \
125125
--model-or-model-name-or-path Qwen/Qwen2.5-1.5B-Instruct \
126126
--output-file outputs/preds.jsonl \
127127
--batch-size 8 \
@@ -163,7 +163,7 @@ <h2 id="citation">📄 Citation</h2>
163163
<pre><code>@inproceedings{llmsql_bench,
164164
title={LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL},
165165
author={Pihulski, Dzmitry and Charchut, Karol and Novogrodskaia, Viktoria and Koco{'n}, Jan},
166-
booktitle={2025 IEEE ICувцDMW},
166+
booktitle={2025 IEEE International Conference on Data Mining Workshops (ICDMW)},
167167
year={2025},
168168
organization={IEEE}
169169
}

‎docs/docs/inference.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,12 @@ Inference API Reference
1414

1515
---
1616

17+
.. automodule:: llmsql.inference.inference_api
18+
:members:
19+
:undoc-members:
20+
21+
---
22+
1723
.. raw:: html
1824

1925
<div style="text-align:center; margin-top:2rem; color:#666;">

‎docs/docs/usage.rst‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,41 @@ Using vllm backend.
7777
print(report)
7878
7979
80+
Using OpenAI-compateble API.
81+
82+
.. code-block:: python
83+
84+
from llmsql import inference_api
85+
from dotenv import load_dotenv
86+
import os
87+
load_dotenv()
88+
89+
# Run inference (will take some time)
90+
results = inference_api(
91+
model_name="gpt-5-mini",
92+
base_url="https://api.openai.com/v1/",
93+
api_key=os.environ["OPENAI_API_KEY"],
94+
api_kwargs={
95+
"response_format": {
96+
"type": "text"
97+
},
98+
"verbosity": "medium",
99+
"reasoning_effort": "medium",
100+
"store": False
101+
},
102+
requests_per_minute=100,
103+
output_file="test_output_api.jsonl",
104+
limit=50,
105+
num_fewshots = 5,
106+
seed=42,
107+
version="2.0"
108+
)
109+
110+
# Evaluate the results
111+
evaluator = LLMSQLEvaluator()
112+
report = evaluator.evaluate(outputs_path="outputs/preds_transformers.jsonl")
113+
print(report)
114+
80115
---
81116

82117
.. raw:: html

‎examples/inference_api.ipynb‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": 1,
6+
"id": "5409b21a",
7+
"metadata": {},
8+
"outputs": [
9+
{
10+
"data": {
11+
"text/plain": [
12+
"True"
13+
]
14+
},
15+
"execution_count": 1,
16+
"metadata": {},
17+
"output_type": "execute_result"
18+
}
19+
],
20+
"source": [
21+
"from llmsql import inference_api\n",
22+
"from dotenv import load_dotenv\n",
23+
"import os\n",
24+
"load_dotenv()"
25+
]
26+
},
27+
{
28+
"cell_type": "code",
29+
"execution_count": 2,
30+
"id": "581e9c25",
31+
"metadata": {},
32+
"outputs": [
33+
{
34+
"name": "stderr",
35+
"output_type": "stream",
36+
"text": [
37+
"2026-03-04 08:10:34,504 [INFO] llmsql-bench: Removing existing path: llmsql_workdir/questions.jsonl\n",
38+
"2026-03-04 08:10:34,506 [INFO] llmsql-bench: Downloading questions.jsonl from Hugging Face Hub...\n"
39+
]
40+
},
41+
{
42+
"data": {
43+
"application/vnd.jupyter.widget-view+json": {
44+
"model_id": "a71443d8f32840838ba484eadf26d9d0",
45+
"version_major": 2,
46+
"version_minor": 0
47+
},
48+
"text/plain": [
49+
"questions.jsonl: 0%| | 0.00/18.3M [00:00<?, ?B/s]"
50+
]
51+
},
52+
"metadata": {},
53+
"output_type": "display_data"
54+
},
55+
{
56+
"name": "stderr",
57+
"output_type": "stream",
58+
"text": [
59+
"2026-03-04 08:10:35,608 [INFO] llmsql-bench: Downloaded questions.jsonl to: llmsql_workdir/questions.jsonl\n",
60+
"2026-03-04 08:10:35,608 [INFO] llmsql-bench: Removing existing path: llmsql_workdir/tables.jsonl\n",
61+
"2026-03-04 08:10:35,611 [INFO] llmsql-bench: Downloading tables.jsonl from Hugging Face Hub...\n"
62+
]
63+
},
64+
{
65+
"data": {
66+
"application/vnd.jupyter.widget-view+json": {
67+
"model_id": "62ec9ecc8d8b48f7a835019688ee1894",
68+
"version_major": 2,
69+
"version_minor": 0
70+
},
71+
"text/plain": [
72+
"tables.jsonl: 0%| | 0.00/45.3M [00:00<?, ?B/s]"
73+
]
74+
},
75+
"metadata": {},
76+
"output_type": "display_data"
77+
},
78+
{
79+
"name": "stderr",
80+
"output_type": "stream",
81+
"text": [
82+
"2026-03-04 08:10:36,293 [INFO] llmsql-bench: Downloaded tables.jsonl to: llmsql_workdir/tables.jsonl\n",
83+
"Generating: 100%|██████████| 50/50 [00:33<00:00, 1.48it/s]\n",
84+
"2026-03-04 08:11:11,394 [INFO] llmsql-bench: Generation completed. 50 results saved to test_output_api.jsonl\n"
85+
]
86+
}
87+
],
88+
"source": [
89+
"results = inference_api(\n",
90+
" model_name=\"gpt-5-mini\",\n",
91+
" base_url=\"https://api.openai.com/v1/\",\n",
92+
" api_key=os.environ[\"OPENAI_API_KEY\"],\n",
93+
" api_kwargs={\n",
94+
" \"response_format\": {\n",
95+
" \"type\": \"text\"\n",
96+
" },\n",
97+
" \"verbosity\": \"medium\",\n",
98+
" \"reasoning_effort\": \"medium\",\n",
99+
" \"store\": False\n",
100+
" },\n",
101+
" requests_per_minute=100,\n",
102+
" output_file=\"test_output_api.jsonl\",\n",
103+
" limit=50,\n",
104+
" num_fewshots = 5,\n",
105+
" seed=42,\n",
106+
" version=\"2.0\"\n",
107+
")"
108+
]
109+
}
110+
],
111+
"metadata": {
112+
"kernelspec": {
113+
"display_name": "llmsql-benchmark-3.11 (3.11.13)",
114+
"language": "python",
115+
"name": "python3"
116+
},
117+
"language_info": {
118+
"codemirror_mode": {
119+
"name": "ipython",
120+
"version": 3
121+
},
122+
"file_extension": ".py",
123+
"mimetype": "text/x-python",
124+
"name": "python",
125+
"nbconvert_exporter": "python",
126+
"pygments_lexer": "ipython3",
127+
"version": "3.11.13"
128+
}
129+
},
130+
"nbformat": 4,
131+
"nbformat_minor": 5
132+
}

‎examples/test_output_api.jsonl‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
{"question_id": 15, "completion": "SELECT \"Launched\" FROM \"Table\" WHERE \"Laid down\" = \"September 1, 1964\";"}
2+
{"question_id": 48, "completion": "SELECT \"Tone (Latvian notation: /~/ - level, /^/ - broken)\" FROM \"Table\" WHERE \"Translation\" = \"Gen.Sing. plague\";"}
3+
{"question_id": 4, "completion": "SELECT COUNT(\"School/Club Team\") FROM \"Table\" WHERE \"Player\" = \"Jalen Rose\";"}
4+
{"question_id": 21, "completion": "SELECT \"US Air Force equivalent\" FROM \"Table\" WHERE \"Commonwealth equivalent\" = \"Pilot Officer\";"}
5+
{"question_id": 31, "completion": "SELECT COUNT(\"Total W–L\") FROM \"Table\" WHERE \"Doubles W–L\" = \"11–11\";"}
6+
{"question_id": 42, "completion": "SELECT \"BTs retail price (regulated)\" FROM \"Table\" WHERE \"Tariff code\" = \"ff0 PRS\";"}
7+
{"question_id": 7, "completion": "SELECT \"Date\" FROM \"Table\" WHERE \"Circuit\" = \"Misano\";"}
8+
{"question_id": 28, "completion": "SELECT \"Rank Each wrestlers total number of days as champion are ranked highest to lowest; wrestlers with the same number mean that they are tied for that certain rank.\" FROM \"Table\" WHERE \"Wrestler\" = \"Go Shiozaki\";"}
9+
{"question_id": 36, "completion": "SELECT \"Frequency\" FROM \"Table\" WHERE \"Market/Rank\" = \"Burlington - Plattsburgh , Vermont - New York /143\";"}
10+
{"question_id": 23, "completion": "SELECT \"Rank in Spanish\" FROM \"Table\" WHERE \"Rank in English\" = \"Major\";"}
11+
{"question_id": 11, "completion": "SELECT COUNT(DISTINCT \"Nationality\") FROM \"Table\" WHERE \"NHL team\" = \"New Jersey Devils\";"}
12+
{"question_id": 47, "completion": "SELECT \"BTs retail price (regulated)\" FROM \"Table\" WHERE \"Tariff code\" = \"g10\";"}
13+
{"question_id": 12, "completion": "SELECT \"Pick\" FROM \"Table\" WHERE \"Player\" = \"Dorain Anneck\";"}
14+
{"question_id": 16, "completion": "SELECT \"#\" FROM \"Table\" WHERE \"Commissioned\" = \"December 18, 1965\";"}
15+
{"question_id": 27, "completion": "SELECT \"Combined days\" FROM \"Table\" WHERE \"Wrestler\" = \"Go Shiozaki\";"}
16+
{"question_id": 32, "completion": "SELECT COUNT(\"Singles W–L\") FROM \"Table\" WHERE \"Doubles W–L\" = \"11–14\";"}
17+
{"question_id": 22, "completion": "SELECT \"Commonwealth equivalent\" FROM \"Table\" WHERE \"US Air Force equivalent\" = \"Major General\";"}
18+
{"question_id": 43, "completion": "SELECT \"Approx premium\" FROM \"Table\" WHERE \"Tariff code\" = \"g9\";"}
19+
{"question_id": 49, "completion": "SELECT MIN(\"Radius (R ☉ )\") FROM \"Table\";"}
20+
{"question_id": 34, "completion": "SELECT MAX(\"Ties played\") FROM \"Table\" WHERE \"Player\" = \"Josip Palada Category:Articles with hCards\";"}
21+
{"question_id": 39, "completion": "SELECT \"Format\" FROM \"Table\" WHERE \"Branding\" = \"1290 WKBK W281AU 104.1\";"}
22+
{"question_id": 6, "completion": "SELECT \"No\" FROM \"Table\" WHERE \"Race winner\" = \"Kevin Curtain\";"}
23+
{"question_id": 18, "completion": "SELECT \"Laid down\" FROM \"Table\" WHERE \"Commissioned\" = \"October 29, 1965\";"}
24+
{"question_id": 13, "completion": "SELECT \"Nationality\" FROM \"Table\" WHERE \"NHL team\" = \"Vancouver Canucks\";"}
25+
{"question_id": 38, "completion": "SELECT \"Branding\" FROM \"Table\" WHERE \"Calls\" = \"WRKO\";"}
26+
{"question_id": 2, "completion": "SELECT \"School/Club Team\" FROM \"Table\" WHERE \"Years in Toronto\" = \"1995-96\";"}
27+
{"question_id": 29, "completion": "SELECT \"Province\" FROM \"Table\" WHERE \"Electorate\" = \"Grey and Bell\";"}
28+
{"question_id": 44, "completion": "SELECT COUNT(\"Tariff code\") FROM \"Table\" WHERE \"BTs retail price (regulated)\" = \"2p/min or inclusive\";"}
29+
{"question_id": 26, "completion": "SELECT \"Rank Each wrestlers total number of days as champion are ranked highest to lowest; wrestlers with the same number mean that they are tied for that certain rank.\" FROM \"Table\" WHERE \"Wrestler\" = \"Bryan Danielson\";"}
30+
{"question_id": 33, "completion": "SELECT \"Total W–L\" FROM \"Table\" WHERE \"Player\" = \"Boro Jovanović Category:Articles with hCards\";"}
31+
{"question_id": 37, "completion": "SELECT \"Branding\" FROM \"Table\" WHERE \"Group owner\" = \"Qantam of Cape Cod, LLC\";"}
32+
{"question_id": 8, "completion": "SELECT COUNT(DISTINCT \"Position\") FROM \"Table\" WHERE \"College/junior/club team\" = \"Sherbrooke Faucons (QMJHL)\";"}
33+
{"question_id": 3, "completion": "SELECT \"School/Club Team\" FROM \"Table\" WHERE \"Years in Toronto\" = \"2003-06\";"}
34+
{"question_id": 24, "completion": "SELECT \"Wrestler\" FROM \"Table\" WHERE \"# of reigns\" = 2;"}
35+
{"question_id": 14, "completion": "SELECT \"Pick\" FROM \"Table\" WHERE \"College/junior/club team\" = \"Springfield Olympics (NEJHL)\";"}
36+
{"question_id": 45, "completion": "SELECT COUNT(\"Tariff code\") FROM \"Table\" WHERE \"BTs retail price (regulated)\" = \"2.553p/min\";"}
37+
{"question_id": 30, "completion": "SELECT \"Province\" FROM \"Table\" WHERE \"Electorate\" = \"Bay of Islands\";"}
38+
{"question_id": 25, "completion": "SELECT MIN(\"# of reigns\") FROM \"Table\";"}
39+
{"question_id": 19, "completion": "SELECT \"Commonwealth equivalent\" FROM \"Table\" WHERE \"Rank in Spanish\" = \"Coronel\";"}
40+
{"question_id": 40, "completion": "SELECT \"Market/Rank\" FROM \"Table\" WHERE \"Calls\" = \"WCRN\";"}
41+
{"question_id": 35, "completion": "SELECT SUM(\"Ties played\") FROM \"Table\" WHERE \"Total W–L\" = \"38–24\";"}
42+
{"question_id": 50, "completion": "SELECT \"Spectral type\" FROM \"Table\" WHERE \"Star (Pismis24-#)\" = \"1SW\";"}
43+
{"question_id": 20, "completion": "SELECT \"Rank in Spanish\" FROM \"Table\" WHERE \"Rank in English\" = \"Group Captain\";"}
44+
{"question_id": 1, "completion": "SELECT \"Nationality\" FROM \"Table\" WHERE \"Player\" = \"Terrence Ross\";"}
45+
{"question_id": 5, "completion": "SELECT \"Circuit\" FROM \"Table\" WHERE \"Round\" = \"Assen\";"}
46+
{"question_id": 10, "completion": "SELECT COUNT(DISTINCT \"College/junior/club team\") FROM \"Table\" WHERE \"NHL team\" = \"Washington Capitals\";"}
47+
{"question_id": 46, "completion": "SELECT \"Prefixes\" FROM \"Table\" WHERE \"Scheme\" = \"Pence per minute, fixed at all times\" AND \"Approx premium\" = \"3p/min\";"}
48+
{"question_id": 17, "completion": "SELECT \"#\" FROM \"Table\" WHERE \"Commissioned\" = \"September 30, 1967\";"}
49+
{"question_id": 9, "completion": "SELECT \"Nationality\" FROM \"Table\" WHERE \"College/junior/club team\" = \"Thunder Bay Flyers (USHL)\";"}
50+
{"question_id": 41, "completion": "SELECT \"Frequency\" FROM \"Table\" WHERE \"Calls\" = \"WEGP\";"}

‎llmsql/__init__.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,12 @@ def __getattr__(name: str): # type: ignore
2626
from .inference.inference_transformers import inference_transformers
2727

2828
return inference_transformers
29+
elif name == "inference_api":
30+
from .inference.inference_api import inference_api
31+
32+
return inference_api
33+
2934
raise AttributeError(f"module {__name__} has no attribute {name!r}")
3035

3136

32-
__all__ = ["evaluate", "inference_vllm", "inference_transformers"]
37+
__all__ = ["evaluate", "inference_vllm", "inference_transformers", "inference_api"]

0 commit comments

Comments
 (0)