A few-shot table-to-text generation study using Qwen3-4B on the ToTTo dataset.
The project investigates how different prompting and table-marking strategies affect the ability of a large language model to generate fluent, factual natural-language descriptions from structured tables.
Generated sentences are evaluated using BLEU and BLEURT.
The project was developed as part of CENG 467 — Natural Language Processing at İzmir Institute of Technology.
Table-to-text generation is the task of converting structured tabular information into fluent natural-language descriptions.
A successful system should generate text that:
- reflects the relevant information in the table,
- remains factually consistent with the provided data,
- avoids unsupported information,
- and produces fluent natural-language output.
This project explores table-to-text generation using Qwen3-4B and the ToTTo dataset.
Three experimental prompting strategies are evaluated.
The experiments investigate:
- few-shot prompting,
- highlighted-cell guidance,
- stricter factuality instructions,
- row-level importance markers,
- structured auxiliary context,
- BLEU evaluation,
- and BLEURT evaluation.
The project uses the ToTTo table-to-text dataset through:
GEM/totto
The validation split is used for the experiments.
Each example contains structured information including:
- a table,
- highlighted cells,
- page title,
- section title,
- section text,
- and a target natural-language sentence.
For the experiments in this repository, the first:
300 validation examples
are evaluated.
The objective is to generate a single fluent sentence describing the relevant information in a table.
The general workflow is:
ToTTo Table
│
▼
Highlighted Cells
│
▼
Table Formatting
│
▼
Few-Shot Prompt
│
▼
Qwen3-4B
│
▼
Generated Sentence
│
▼
BLEU & BLEURT Evaluation
Auxiliary context such as the page and section titles can also be provided to help identify the subject of the table.
All three experiments use:
Qwen/Qwen3-4B
The model is loaded using Hugging Face Transformers.
The implementation automatically uses a CUDA-compatible GPU when available.
Model: Qwen3-4B
Dataset: GEM/totto
Dataset Split: validation
Number of Examples: 300
Maximum Input Length: 4096 tokens
Maximum New Tokens: 50
Three prompt configurations are evaluated.
The first experiment establishes the primary prompting setup.
Highlighted cells in each ToTTo table are wrapped using:
<highlight>
...
</highlight>
The model is instructed to:
- focus on highlighted information,
- produce a fluent factual sentence,
- use auxiliary context only when necessary,
- avoid hallucinating information,
- and return exactly one sentence.
The prompt also contains few-shot examples demonstrating the expected table-to-text behavior.
Implementation:
CENG467_project2_code_03/CENG467_Project2_Q1.py
Prompt:
CENG467_project2_code_03/prompt_Q1.txt
Results:
CENG467_project2_code_03/Q1_outputs.json
BLEU: 19.79
BLEURT: 0.551
This configuration achieved the strongest BLEU and BLEURT scores among the three evaluated experiments.
The second experiment investigates whether stronger instructions about factuality improve table-to-text generation.
The model is explicitly assigned the role of a:
strict Wikipedia data editor
The prompt emphasizes that the model should:
- use only highlighted cells as factual evidence,
- avoid non-highlighted information,
- avoid inference,
- avoid hallucination,
- produce neutral encyclopedic text,
- and output exactly one sentence.
The underlying table representation remains similar to Experiment 1.
Implementation:
CENG467_project2_code_03/CENG467_Project2_Q2_g.py
Prompt:
CENG467_project2_code_03/prompt_Q2_g.txt
Results:
CENG467_project2_code_03/Q2_g_outputs.json
BLEU: 16.77
BLEURT: 0.544
Despite the stronger factuality constraints, this configuration produced a lower BLEU score than Experiment 1.
Its BLEURT score remained relatively close to the first experiment.
The third experiment modifies the table representation itself.
In addition to highlighted cells, rows containing relevant information are marked using:
<important>
...
</important>
Highlighted values within those rows continue to use:
<highlight>
...
</highlight>
This provides the model with two levels of structural guidance:
Important Row
│
└── Highlighted Cell
The goal is to investigate whether explicitly indicating relevant rows helps the model identify useful table context.
Implementation:
CENG467_project2_code_03/CENG467_Project2_Q3.py
Prompt:
CENG467_project2_code_03/prompt_Q3.txt
Results:
CENG467_project2_code_03/Q3_outputs.json
BLEU: 18.16
BLEURT: 0.522
The important-row representation improved BLEU relative to Experiment 2 but did not outperform the original Q1 configuration.
The three experiments produced the following scores:
| Experiment | Prompt Strategy | BLEU | BLEURT |
|---|---|---|---|
| Q1 | Highlight-based few-shot prompting | 19.79 | 0.551 |
| Q2-g | Strict factuality instructions | 16.77 | 0.544 |
| Q3 | Highlight + important-row markers | 18.16 | 0.522 |
Among the evaluated configurations, Experiment 1 achieved the highest score on both metrics.
The experiments suggest several observations.
Experiment 2 introduced much stricter factuality constraints than Experiment 1.
However, BLEU decreased from:
19.79
to:
16.77
This suggests that simply adding more restrictive instructions does not necessarily improve similarity to the target references.
Experiment 3 introduced explicit <important> markers for rows containing highlighted cells.
Its BLEU score:
18.16
was higher than Experiment 2 but lower than the original Q1 setup.
This suggests that table representation itself can influence model generation behavior.
BLEU primarily measures lexical overlap with reference sentences.
BLEURT uses a learned evaluation model and can capture semantic similarity beyond exact word overlap.
Using both metrics provides a broader view of the generated text than relying on BLEU alone.
Each JSON result file contains:
{
"metrics": {
"bleu": "...",
"bleurt": "..."
},
"examples": [
{
"example_id": 0,
"generated": "...",
"target": "..."
}
]
}This makes it possible to inspect both the overall quantitative results and individual model generations.
BLEU is calculated using:
sacrebleu
at corpus level.
It measures n-gram overlap between generated sentences and reference sentences.
BLEURT evaluation uses:
lucadiliello/BLEURT-20
The BLEURT score provides a learned semantic evaluation of the generated sentences relative to their reference targets.
qwen-table-to-text-generation/
│
├── CENG467_project2_code_03/
│ │
│ ├── CENG467_Project2_Q1.py
│ ├── CENG467_Project2_Q2_g.py
│ ├── CENG467_Project2_Q3.py
│ │
│ ├── prompt_Q1.txt
│ ├── prompt_Q2_g.txt
│ ├── prompt_Q3.txt
│ │
│ ├── Q1_outputs.json
│ ├── Q2_g_outputs.json
│ └── Q3_outputs.json
│
├── CENG467_project2_report_G03.pdf
│
└── README.md
Clone the repository:
git clone https://github.com/behicekadioglu/qwen-table-to-text-generation.gitEnter the code directory:
cd qwen-table-to-text-generation/CENG467_project2_code_03Run Experiment 1:
python CENG467_Project2_Q1.pyRun Experiment 2:
python CENG467_Project2_Q2_g.pyRun Experiment 3:
python CENG467_Project2_Q3.pyEach experiment loads the ToTTo validation dataset, generates sentences for 300 examples, evaluates them using BLEU and BLEURT, and saves the results to its corresponding JSON file.
The implementation uses libraries including:
torch
transformers
datasets
sacrebleu
bleurt-pytorch
The project relies on Hugging Face for both the Qwen model and the ToTTo dataset interface.
Running Qwen3-4B locally may require substantial memory, and a CUDA-compatible GPU is recommended.
- Table-to-Text Generation
- Natural Language Generation
- Few-Shot Learning
- Prompt Engineering
- Qwen3-4B
- Hugging Face Transformers
- BLEU
- BLEURT
- Python
- PyTorch
- Hugging Face Datasets
Several limitations should be considered when interpreting the results.
The experiments evaluate the first 300 examples from the ToTTo validation split rather than the entire dataset.
Therefore, the reported scores should be interpreted as results for this experimental subset.
All three experiments use Qwen3-4B.
The study therefore focuses primarily on prompt and table-representation changes rather than a broad model comparison.
BLEU and BLEURT provide useful quantitative signals but cannot fully evaluate:
- factual consistency,
- hallucination,
- fluency,
- completeness,
- or human preference.
A more comprehensive evaluation could include manual assessment or additional factuality metrics.
The experiments demonstrate that relatively small changes in instructions and table representation can affect generation results.
Further experiments would be needed to determine whether the observed differences generalize beyond the evaluated subset.
A detailed report describing the project, methodology, experiments, and results is included in:
CENG467_project2_report_G03.pdf
This project was developed for CENG 467 — Natural Language Processing at İzmir Institute of Technology.
The purpose of the project was to investigate table-to-text generation with large language models and experimentally compare different prompting and structured-input strategies.
The repository is presented as an academic NLP study.
This project was completed collaboratively by:
- Behice Kadıoğlu
- Zeynep Naz Ödenir
Both authors contributed to the project as part of the CENG 467 group coursework.
Computer Engineering
İzmir Institute of Technology
GitHub: @behicekadioglu
Project collaborator
The project uses external resources including:
- the ToTTo dataset,
- the GEM dataset interface,
- Qwen3-4B,
- and BLEURT-20.
These resources belong to their respective authors and organizations.
Please refer to their original documentation and publications for citation and licensing information.