Verifiable Explanations Generation for LLM Reasoning

This repository contains the implementation and evaluation artifacts for the Bachelor thesis project:

Verifiable Explanations Generation for LLM Reasoning

The project implements a structured pipeline for generating reasoning traces from a Large Language Model (LLM) and verifying them using deterministic symbolic logic within a restricted propositional reasoning domain.

---

Repository Structure

code/
    Full runnable implementation code.

data/
    Final dataset workbook used for testing.

results/
    Final runtime logging workbook containing runtime logs, metrics, visuals, observations, and discussion.

requirements.txt
    Python dependencies needed to run the project.

README.md
    Repository documentation.

---

Project Pipeline

The implemented pipeline follows the structure below:

Raw Input
→ Normalizer
→ Normalized Prompt Parser
→ LLM Reasoning Module
→ LLM Response Parser
→ Translator
→ Verifier

The pipeline receives a natural-language reasoning prompt, normalizes it into a controlled format, generates a structured LLM reasoning trace, translates the problem and trace into symbolic logic, and verifies each reasoning step using deterministic inference rules.

---

Main Components

The implementation is organized into the following main modules:

code/normalizer/
    Normalizes raw input and applies the N1–N8 normalization pipeline.

code/parsers/
    Contains the normalized prompt parser and LLM response parser.

code/llm_response/
    Generates structured reasoning traces using the configured LLM.

code/translator/
    Translates parsed natural-language structures into symbolic logic.

code/verifier/
    Verifies symbolic reasoning traces using deterministic inference rules.

code/repair/
    Implements the global user-guided repair loop.

code/tests/
    Contains automated and manual test scripts used during the testing phase.

---

Dataset

The final dataset is available at:

data/Dataset.xlsx

It contains:

• D1: 120 full-pipeline examples.
• D2: 30 intentional-rejection examples.

D1 examples were designed to pass through the full reasoning pipeline, while D2 examples were designed to test intentional rejection behavior at the Normalizer or Verifier level.

---

Runtime Logging and Results

The final runtime logging and post-running analysis workbook is available at:

results/Runtime_Logging_Table.xlsx

It contains the following sheets:

• Runtime_Logs
• Metrics Design
• Metrics Calculation
• Visuals
• Observation
• Discussion

This workbook is the main evidence base used for the evaluation and results chapter of the thesis.

---

Requirements

Install the required Python dependencies using:

pip install -r requirements.txt

The project also requires local Ollama models for LLM-based generation.

The configured model names are defined in:

code/config.py

Current configuration:

NORMALIZER_LLM_MODEL = qwen2.5:14b
GENERATION_LLM_MODEL = llama3.1:8b

---

Running the Project

From the repository root, move into the code folder:

cd code

Run the final automated test:

python tests/test_Final_a.py

Run the final manual test:

python tests/test_Final_b.py

Run the repair-loop automated test:

python tests/test_repair_a.py

Run the repair-loop manual test:

python tests/test_repair_b.py

---

Notes

This repository is provided as an external artifact for the Bachelor thesis.

It contains the final runnable code, dataset, and runtime logging workbook used during the implementation, testing, and evaluation phases.

After pasting it, press:

```text
Ctrl + S