Fix evaluator class reference and add prompt template - #69
Merged
Conversation
Updated evaluator class reference and added prompt template section with examples.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
DzmitryPihulski
requested changes
Feb 7, 2026
| output_file="path_to_your_outputs.jsonl", | ||
| - Use [`llmsql.inference_transformers`](./llmsql/inference/inference_transformers.py) (the function for transformers inference) for generation of SQL predictions with your model. If you want to do vllm based inference, use [`llmsql.inference_vllm`](./llmsql/inference/inference_vllm.py). Works both with HF model id, e.g. `Qwen/Qwen2.5-1.5B-Instruct` and model instance passed directly, e.g. `inference_transformers(model_or_model_name_or_path=model, ...)` | ||
| - Evaluate results against the benchmark with the [`llmsql.LLMSQLEvaluator`](./llmsql/evaluation/evaluator.py) evaluator class. | ||
| - Evaluate results against the benchmark with the [`llmsql.evaluate`](./llmsql/evaluation/evaluator.py) evaluator class. |
Collaborator
There was a problem hiding this comment.
evaluate() is a function, not evaluator class
|
|
||
| ## Prompt Template | ||
|
|
||
| The prompt defines explicit constraints on the generated output. The model is instructed to output only a valid SQL `SELECT` query, to use a fixed table name (`"Table"`), to quote all table and column names, and to restrict generation to the specified SQL functions, condition operators, and keywords. The full prompt specification is provided in the prompt template. |
Collaborator
There was a problem hiding this comment.
Add that "Table" will be replaced with actual table name in process of evaluation.
The "Table" word is used just to not confuse the model with the table names.
Comment on lines
+167
to
+200
| ```python | ||
| def build_prompt_1shot( | ||
| question: str, | ||
| headers: list[str], | ||
| types: list[str], | ||
| sample_row: list[str | float | int], | ||
| ) -> str: | ||
| return f"""You are an expert SQLite SQL query generator. | ||
| Your task: Given a question and a table schema, output ONLY a valid SQL SELECT query. | ||
| ⚠️ STRICT RULES: | ||
| - Output ONLY SQL (no explanations, no markdown, no ``` fences) | ||
| - Use table name "Table" | ||
| - Allowed functions: ['MAX', 'MIN', 'COUNT', 'SUM', 'AVG'] | ||
| - Allowed condition operators: ['=', '>', '<', '!='] | ||
| - Allowed SQL keywords: ['SELECT', 'WHERE', 'AND'] | ||
| - Always use "" with all column names and table name | ||
|
|
||
| ### EXAMPLE 1: | ||
| Question: What is the price of the Samsung Galaxy S23? | ||
| Columns: ['Brand', 'Model', 'Price', 'Storage', 'Color'] | ||
| Types: ['text', 'text', 'real', 'text', 'text'] | ||
| Sample row: ['Apple', 'iPhone 14', 899.99, '128GB', 'White'] | ||
| SQL: SELECT "Price" FROM "Table" WHERE "Brand" = "Samsung" AND "Model" = "Galaxy S23"; | ||
|
|
||
| ### NOW ANSWER: | ||
| Question: {question} | ||
| Columns: {headers} | ||
| Types: {types} | ||
| Sample row: {sample_row} | ||
| SQL:""" | ||
| ``` | ||
|
|
||
| Implementations of 0-shot, 1-shot, and 5-shot prompt templates are available here: | ||
| 👉 [link-to-file](./llmsql/prompts/prompts.py) |
Collaborator
There was a problem hiding this comment.
Here we do not need a function, we need a text (prompt it self).
It is also better to add 5 shot version, not 1 shot.
DzmitryPihulski
added
documentation
Updated README
Collaborator
Author
|
README updated according to reviewer’s suggestions |
DzmitryPihulski
approved these changes
Feb 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated evaluator class reference and added prompt template section with examples.