Prompt Sculpting: A Research Implementation

This repository contains the experimental code and data for evaluating Prompt Sculpting, a novel prompting technique that combines constraint-based pruning with programmatic scaffolding to improve reasoning in Large Language Models.

Overview

This research investigates whether explicitly constraining LLMs to use only problem-specific information (pruning external knowledge) combined with step-by-step reasoning scaffolds (scaffolding) can improve performance on logic and mathematical reasoning tasks.

Three Prompting Methods Tested

1. Zero-Shot: Direct question with no additional guidance
2. Scaffolding: Chain-of-Thought style step-by-step reasoning prompts
3. Sculpting: Constraint-based pruning + scaffolding (the proposed method)

The sculpting prompt explicitly instructs the model to:

• Use ONLY information given in the problem
• NOT use outside common sense or real-world knowledge
• Break down reasoning step-by-step
• State the final answer clearly

Repository Structure

.
├── run_experiment.py                          # Logic puzzles experiment (5 puzzles, gpt-4o-mini)
├── run_gsm8k_gpt4o.py                              # GSM8K experiment with GPT-4o (100 samples)
├── run_gsm8k_gpt5.py                               # GSM8K experiment with GPT-5 (100 samples)
├── run_gsm8k_gpt5_full_benchmark.py                # Full GSM8K benchmark (1400 samples, GPT-5)
├── re_evaluate_results.py                        # Initial re-evaluation script
├── re_evaluate_gpt4o.py            # Final robust re-evaluation for GPT-4o
├── experiment_results.json                     # Logic puzzles results
├── gsm8k_experiment_results.json               # GSM8K results (GPT-4o-mini, initial)
├── gsm8k_gpt4o_results.json                    # GSM8K results (GPT-4o)
├── gsm8k_gpt4o_results_CORRECTED.json         # Corrected GPT-4o results
├── gsm8k_gpt5_results.json                     # GSM8K results (GPT-5, 100 samples)
├── gsm8k_gpt5_full_benchmark_results.json     # Full benchmark results (GPT-5, 1400 samples)
├── requirements.txt
└── README.md

Installation

Prerequisites

• Python 3.8+
• OpenAI API key with access to GPT models

Setup

1. Clone the repository:

git clone https://github.com/strongSoda/prompt-sculpting.git
cd prompt-sculpting

2. Install dependencies:

pip install -r requirements.txt

3. Set your OpenAI API key:

# Linux/macOS
export OPENAI_API_KEY='your_key_here'

# Windows
set OPENAI_API_KEY='your_key_here'

# Or use a .env file
echo "OPENAI_API_KEY=your_key_here" > .env

Usage

Logic Puzzles Experiment

Run the logic puzzles experiment (5 puzzles, 3 methods):

python run_experiment.py

Output: experiment_results.json with detailed results and accuracy metrics.

GSM8K Math Experiments

GPT-4o (100 samples)

python run_gsm8k_gpt4o.py

GPT-5 (100 samples)

python run_gsm8k_gpt5.py

GPT-5 Full Benchmark (1400 samples)

python run_gsm8k_gpt5_full_benchmark.py

Note: The full benchmark script supports resuming from interruptions. If stopped, simply re-run the script and it will continue from where it left off.

Re-evaluation Scripts

Due to initial challenges in accurately extracting numerical answers from LLM outputs, we developed increasingly robust evaluation functions. To re-evaluate existing results with the improved extraction logic:

# For GSM8K (initial version)
python re_evaluate_results.py

# For GPT-4o results (final robust version)
python re_evaluate_gpt4o.py

Evaluation Methodology

Answer Extraction Logic

The final evaluation function (extract_final_answer) prioritizes answer extraction in this order:

1. "Final Answer:" marker: Extracts the first number following this prefix
2. LaTeX boxed notation: Extracts numbers from \boxed{...}
3. Fallback: Extracts the last number in the entire output

This approach handles various response formats including those with intermediate calculations, units, currency symbols, and comma separators.

Why Re-evaluation Was Needed

Initial evaluation scripts had limitations in:

• Handling various number formats (currency, commas, decimals)
• Distinguishing between intermediate calculations and final answers
• Parsing answers not explicitly marked with "Final Answer:"

The final evaluation function was iteratively improved to be more robust, requiring re-evaluation of early experimental results to ensure accuracy.

Configuration

Model Selection

Edit the MODEL variable in each script:

• run_experiment.py: "gpt-4o-mini" (default)
• run_gsm8k_gpt4o.py: "gpt-4o"
• run_gsm8k_gpt5.py: "gpt-5"
• run_gsm8k_gpt5_full_benchmark.py: "gpt-5"

Sample Size

For GSM8K experiments, adjust the SAMPLE_SIZE variable:

SAMPLE_SIZE = 100  # Number of problems to evaluate

Temperature

All experiments use temperature=0.0 for deterministic outputs (except noted otherwise in code).

Results Format

Results are saved as JSON files with the following structure:

[
  {
    "puzzle_id": "gsm8k_test_0",
    "question": "...",
    "expected_answers": ["42"],
    "outputs": {
      "zero_shot": {
        "prompt": "...",
        "output": "...",
        "is_correct": true
      },
      "scaffolding": { ... },
      "sculpting": { ... }
    }
  }
]

Key Findings

The experiments evaluate whether prompt sculpting (explicit constraints + scaffolding) improves reasoning compared to zero-shot and standard scaffolding approaches. Results are documented in the accompanying research paper.

Rate Limiting

The scripts include:

• Exponential backoff retry logic for API failures
• Sleep delays between requests (0.5-1 second)
• Incremental saving (every 50 puzzles in full benchmark)

Citation

If you use this code or methodology in your research, please cite:

@misc{khan2025dontneedpromptengineering,
      title={You Don't Need Prompt Engineering Anymore: The Prompting Inversion}, 
      author={Imran Khan},
      year={2025},
      eprint={2510.22251},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.22251}, 
}

License

MIT License - see LICENSE file for details.

Contact

For questions or issues, please open a GitHub issue or contact the author.

Acknowledgments

This research builds upon foundational work in prompt engineering including:

• Chain-of-Thought prompting (Wei et al., 2023)
• Plan-and-Solve prompting (Wang et al., 2023)
• Least-to-Most prompting (Zhou et al., 2023)

The GSM8K dataset is from Cobbe et al. (2021).