Skip to content

README.md

Latest commit

 

History

History
420 lines (322 loc) · 12.6 KB

README.md

File metadata and controls

420 lines (322 loc) · 12.6 KB
Raw

🔍 Code Inspector

Automatic Code Evaluation System Using CodeBERT and Token-based Similarity

A comprehensive research project for evaluating code similarity using state-of-the-art deep learning (CodeBERT) combined with traditional token-based analysis methods.

📋 Table of Contents

🎯 Overview

Code Inspector is an automated code evaluation system designed to assess the similarity between student submissions and reference implementations. It combines:

  1. CodeBERT Analysis: Deep learning-based semantic understanding of code
  2. Token Similarity: Traditional token-based comparison for structural analysis
  3. Combined Scoring: Weighted combination of both methods for accurate evaluation

Research Objective

This system aims to measure the accuracy of combining CodeBERT and token-based similarity methods in evaluating functional equivalence of code without executing it.

✨ Features

  • Dual Evaluation Methods

    • CodeBERT semantic embeddings
    • Token-based similarity (Jaccard, Dice coefficients)

  • Multiple Input Sources

    • Direct code input
    • File upload
    • GitHub repository analysis

  • Comprehensive Reporting

    • HTML reports with visualizations
    • JSON exports for further analysis
    • Text reports for documentation

  • Web Interface

    • User-friendly Flask-based UI
    • Real-time evaluation
    • Batch processing support

  • Accuracy Measurement

    • MAE, RMSE, R² metrics
    • Classification accuracy
    • Error analysis and visualization

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    Code Inspector                        │
├─────────────────────────────────────────────────────────┤
│                                                          │
│  Input Layer                                             │
│  ├─ GitHub Repository Manager                           │
│  ├─ File Upload Handler                                 │
│  └─ Code Preprocessor                                    │
│                                                          │
│  Evaluation Layer                                        │
│  ├─ CodeBERT Evaluator                                  │
│  │   ├─ Model: microsoft/codebert-base                  │
│  │   ├─ Embedding Generation                            │
│  │   └─ Cosine Similarity                               │
│  │                                                       │
│  └─ Token Similarity Evaluator                          │
│      ├─ Tokenization                                     │
│      ├─ Identifier Extraction                           │
│      └─ Jaccard/Dice Similarity                         │
│                                                          │
│  Combination Layer                                       │
│  └─ Score Combiner                                       │
│      ├─ Weighted Average                                 │
│      ├─ Pass/Fail Decision                              │
│      └─ Recommendations                                  │
│                                                          │
│  Output Layer                                            │
│  ├─ Report Generator (HTML/JSON/Text)                   │
│  ├─ Accuracy Calculator                                  │
│  └─ Visualization                                        │
│                                                          │
│  Interface Layer                                         │
│  ├─ Flask Web Application                               │
│  └─ Command-line Interface                              │
└─────────────────────────────────────────────────────────┘

🚀 Installation

Prerequisites

  • Python 3.8 or higher
  • pip package manager
  • Git (for GitHub integration)
  • (Optional) CUDA-capable GPU for faster CodeBERT inference

Step 1: Clone the Repository

git clone https://github.com/yourusername/CodeInspector.git
cd CodeInspector

Step 2: Create Virtual Environment

# Windows
python -m venv venv
venv\Scripts\activate

# Linux/Mac
python3 -m venv venv
source venv/bin/activate

Step 3: Install Dependencies

pip install -r requirements.txt

Note: First installation may take several minutes as it downloads the CodeBERT model (~500MB).

Step 4: Verify Installation

# Test with sample data
python main.py --student-code samples/student_code_high_similarity.py --reference-code samples/reference_code.py

📖 Usage

Web Interface (Recommended)

  1. Start the web server:

    python app.py

  2. Open your browser and navigate to:

    http://localhost:5000

  3. Upload or paste code and click "Evaluate Code"

Command-line Interface

Evaluate Local Files

python main.py \
  --student-code path/to/student.py \
  --reference-code path/to/reference.py \
  --requirements path/to/requirements.txt \
  --language python \
  --output-dir reports \
  --format html

Evaluate GitHub Repository

python main.py \
  --github-url https://github.com/student/project \
  --reference-code path/to/reference.py \
  --language python

Python API

from main import CodeInspector

# Initialize
inspector = CodeInspector(
    codebert_weight=0.6,
    token_weight=0.4,
    pass_threshold=0.7
)

# Evaluate code
results = inspector.evaluate_code(
    student_code="def add(a, b): return a + b",
    reference_code="def add(x, y): return x + y",
    language='python'
)

# Generate report
inspector.generate_report(
    results,
    student_info={'name': 'John Doe'},
    output_format='html'
)

🔬 Research Methodology

Evaluation Process

  1. Preprocessing

    • Code normalization
    • Comment removal (configurable)
    • Whitespace standardization

  2. CodeBERT Analysis

    • Convert code to embeddings using pre-trained CodeBERT
    • Calculate cosine similarity between embeddings
    • Generate semantic similarity score (0-1)

  3. Token Analysis

    • Extract identifiers, keywords, and tokens
    • Calculate Jaccard similarity
    • Generate structural similarity score (0-1)

  4. Score Combination

    • Default: Weighted average (CodeBERT: 60%, Token: 40%)
    • Alternative methods: Average, Max, Min, Harmonic mean

  5. Grading

    • A: ≥90% similarity
    • B: 80-89%
    • C: 70-79%
    • D: 60-69%
    • F: <60%

Accuracy Metrics

The system measures its own accuracy using:

  • Mean Absolute Error (MAE): Average prediction error
  • Root Mean Squared Error (RMSE): Error with penalty for large deviations
  • R² Score: Correlation between predictions and ground truth
  • Classification Accuracy: Pass/fail decision accuracy

📁 Project Structure

CodeInspector/
│
├── main.py                          # Main orchestrator
├── app.py                           # Flask web application
├── requirements.txt                 # Python dependencies
├── README.md                        # This file
│
├── Core Modules/
│   ├── github_manager.py           # GitHub integration
│   ├── code_preprocessor.py        # Code preprocessing
│   ├── codebert_evaluator.py       # CodeBERT evaluation
│   ├── token_similarity_evaluator.py # Token-based evaluation
│   ├── score_combiner.py           # Score combination
│   ├── accuracy_calculator.py      # Accuracy metrics
│   └── report_generator.py         # Report generation
│
├── templates/                       # HTML templates
│   ├── index.html                  # Main page
│   └── report.html                 # Report page
│
├── samples/                         # Sample data
│   ├── reference_code.py
│   ├── student_code_high_similarity.py
│   ├── student_code_medium_similarity.py
│   ├── student_code_low_similarity.py
│   └── requirements.txt
│
├── reports/                         # Generated reports (created at runtime)
├── uploads/                         # Uploaded files (created at runtime)
└── data/                           # Dataset storage (optional)

🔧 Configuration

Adjusting Weights

Edit the initialization in main.py or app.py:

inspector = CodeInspector(
    codebert_weight=0.6,    # CodeBERT importance (0-1)
    token_weight=0.4,       # Token similarity importance (0-1)
    pass_threshold=0.7      # Minimum score to pass (0-1)
)

Combination Methods

Available methods in score_combiner.py:

  • weighted: Custom weights (default)
  • average: Simple average
  • max: Take maximum score
  • min: Take minimum score (conservative)
  • harmonic: Harmonic mean (penalizes low scores)

Supported Languages

Currently supported:

  • Python (.py)
  • Java (.java)
  • JavaScript (.js)
  • C++ (.cpp)
  • C (.c)

To add more languages, extend the preprocessor and tokenizer.

📊 API Reference

CodeInspector Class

class CodeInspector:
    def __init__(self, github_token=None, codebert_weight=0.6,
                 token_weight=0.4, pass_threshold=0.7)

    def evaluate_code(self, student_code, reference_code,
                     requirements=None, language='python',
                     combination_method='weighted') -> Dict

    def evaluate_github_project(self, student_url, reference_code,
                               requirements=None, language='python') -> Dict

    def batch_evaluate(self, student_codes, reference_code,
                      requirements=None, language='python') -> List[Dict]

    def generate_report(self, evaluation_results, student_info=None,
                       requirements=None, output_format='all') -> Dict[str, str]

Web API Endpoints

  • GET /: Home page
  • POST /evaluate: Evaluate code submission
  • GET /report/<eval_id>: View evaluation report
  • GET /download/<eval_id>/<format>: Download report (html/json/text)
  • POST /batch-evaluate: Batch evaluation
  • GET /api/health: Health check

🧪 Testing

Run Sample Evaluations

# Test high similarity
python main.py --student-code samples/student_code_high_similarity.py --reference-code samples/reference_code.py

# Test medium similarity
python main.py --student-code samples/student_code_medium_similarity.py --reference-code samples/reference_code.py

# Test low similarity
python main.py --student-code samples/student_code_low_similarity.py --reference-code samples/reference_code.py

Expected Results

  • High Similarity: 85-95% combined score, Grade A/B
  • Medium Similarity: 60-75% combined score, Grade B/C
  • Low Similarity: 40-55% combined score, Grade C/D

🤝 Contributing

Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests if applicable
  5. Submit a pull request

📄 License

This project is licensed under the MIT License - see LICENSE file for details.

📚 Citation

If you use this project in your research, please cite:

@software{codeinspector2024,
  title={Code Inspector: Automated Code Evaluation Using CodeBERT and Token Similarity},
  author={Your Name},
  year={2024},
  url={https://github.com/yourusername/CodeInspector}
}

🙏 Acknowledgments

  • Microsoft Research for the CodeBERT model
  • HuggingFace for the Transformers library
  • Flask framework for web interface

📞 Support

For questions, issues, or feature requests:

🗺️ Roadmap

Future enhancements:

  • Support for more programming languages
  • Custom model fine-tuning
  • Plagiarism detection
  • Code quality metrics
  • Integration with LMS platforms
  • Real-time collaboration features
  • Advanced visualization dashboards

Built with ❤️ for Computer Science Education