QA to MCQA

Small utility pipeline for converting open-ended QA examples into four-option multiple-choice QA examples.

The pipeline was used to convert datasets such as MedAESQA and MIRIAD into the MCQA format used in the experiments.

Input format

Input files should be JSON lists with at least:

{
  "question": "Question text",
  "reference_answer": "Correct answer text"
}

Usage

Run the scripts in order:

python generate_raw_distractors.py
python expand_distractors.py
python generate_rank_and_build.py
python balance_mcq_distribution.py

Pipeline steps

1. generate_raw_distractors.py generates an initial pool of distractors for each question-answer pair.

2. expand_distractors.py rewrites or expands the raw distractors to make them more plausible and closer in style to the correct answer.

3. generate_rank_and_build.py filters invalid or overly similar candidates, ranks the remaining distractors, selects three of them, and builds four-option MCQ items.

4. balance_mcq_distribution.py reorders the options so that the correct answer labels are approximately balanced across A, B, C, and D.

Output format

The final output is a JSON file containing MCQA examples:

{
  "question": "Question text",
  "choices": [
    "A. Option text",
    "B. Option text",
    "C. Option text",
    "D. Option text"
  ],
  "answer": "B"
}

Paths

The current scripts use hardcoded dataset paths. To process a different dataset, update the input/output constants at the top of the scripts.

Typical file flow:

data/<dataset>/qa/qa_simplified.json
data/<dataset>/tmp/raw_distractors.json
data/<dataset>/tmp/expanded_distractors.json
data/<dataset>/mcqa/imbalanced_mcqa.json
data/<dataset>/mcqa/mcqa.json

Model

The generation, expansion, and ranking stages use meta-llama/Llama-3.3-70B-Instruct through Hugging Face transformers.

Related repository

This utility was used as part of the experiments in:

Biomedical RAG Scaling: biomedical-rag-scaling.

Notes

Generated distractors should be manually inspected when possible, especially in biomedical datasets where an apparently incorrect option may still be partially correct.

The pipeline assumes four answer options. Adapting it to a different number of options requires modifying the MCQ construction and balancing logic.

Datasets derived from external sources remain subject to the licenses and terms of use of the original datasets.

License

The source code and original documentation in this repository are released under the MIT License. See the LICENSE file for details.

This license does not apply to external datasets, model weights, or other third-party resources. Datasets converted with this pipeline, including generated distractors and derived MCQA files, remain subject to the licenses, terms of use, and attribution requirements of their original source datasets.

Citation

The citation for the associated research article will be added once it becomes available.