English / Italiano
Micro-RAG with a Transformer over RDF Triples: A Comparison Between Multitask and Single-Task Learning and Between Constrained and Free-Form Generation
.
├── config.py Configuration
├── main.ipynb Execution notebook
├── requirements.txt Python dependencies
├── src/ Source files
│ ├── data.py Data preprocessing and task preparation
│ ├── tokenizer.py Byte-level BPE tokenizer
│ ├── retrieval.py TF-IDF retriever
│ ├── model.py Encoder-decoder Transformer
│ ├── training.py Model training
│ ├── evaluation.py Metric evaluation and error analysis
│ └── plotting.py Plot and figure generation
├── data/processed/ Processed dataset
├── models/ Trained Transformers
├── results/ Metrics, predictions and error analyses
│ └── figures/ Plots generated during execution
└── deliverables/ Final documentation
├── report.pdf Report
└── presentation.pptx Presentation
The project evaluates a Transformer trained on RDF triples for multiple tasks. Under constrained generation, single-task and multitask learning are compared, while constrained and free-form generation are compared for the multitask model. Both systems use the same greedy autoregressive decoding when generating sequences.
The two main comparisons are:
- Constrained multitask versus constrained single-task
- Constrained multitask versus multitask with free-form generation
The project also compares three baselines:
- Retrieval-only
- Model-only
- Micro-RAG
The pipeline uses the public WebNLG+ 3.0 dataset and includes:
- Triple normalization and canonical serialization
- TF-IDF retrieval with evidence ranking
- Text-to-triples as a consistency check of candidate triples
- Triples-to-text as controlled semantic reconstruction
- Grounded QA through selection of the answer supported by the evidence
- Encoder-decoder Transformer with dimension 256, 3 encoder and decoder layers and 8 attention heads
- Training with AdamW, mixed precision on CUDA and early stopping
- Byte-level BPE tokenizer with a maximum vocabulary of 8,000 tokens, trained on the training set
Execution requires Git, Python 3.12+ and a Jupyter environment are required. A CUDA GPU is recommended for training.
Open main.ipynb from the project root and execute the cells in order. The notebook installs the dependencies, prepares the data, trains the models, computes the metrics, generates the figures and shows error examples. The original dataset is downloaded automatically during the first preprocessing run.
| Baseline evaluation | Result |
|---|---|
| Retrieval Recall@1 | 79.44% |
| Retrieval Recall@3 | 94.07% |
| Retrieval Recall@5 | 97.22% |
| Retrieval-only QA accuracy | 79.44% |
| Model-only QA accuracy | 0.93% |
| Multitask Micro-RAG QA accuracy | 74.63% |
| Single-task Micro-RAG QA accuracy | 74.63% |
| Metric | Constrained generation | Free-form generation |
|---|---|---|
| Exact triple match | 88.59% | 0.00% |
| Entity precision | 94.69% | 1.07% |
| Entity recall | 94.69% | 1.07% |
| Entity F1 | 94.69% | 1.07% |
| Relation precision | 89.73% | 1.43% |
| Relation recall | 89.73% | 1.43% |
| Relation F1 | 89.73% | 1.43% |
| Triples-to-text BLEU | 43.96 | 1.81 |
| Grounded QA accuracy | 74.63% | 1.30% |
| Grounded QA faithfulness | 100% | 3.52% |
Multitask training achieves the same accuracy and faithfulness as single-task training. The retrieval-only baseline remains 4.81 percentage points ahead of Micro-RAG and the rank analysis shows that the system is most effective when the target evidence is ranked first.
The 100% faithfulness of the constrained systems follows from selecting answers contained in the evidence and does not imply that the model learned grounded free-form generation.
The free-form generation approach learns the canonical triple format in the text-to-triples task, but it does not preserve the entities, relations and evidence supplied in the input. Training loss continues to decrease while validation loss remains high and predictions collapse towards frequent entities and answers.