CSO-Bench

CSO-Bench is a benchmark for evaluating whether Large Language Models (LLMs) can capture developers' practical needs in Code Summary Optimization (CSO)—real-world scenarios where developers refine code summaries (docstrings) without changing the underlying code logic.

This repository accompanies the paper:

Can LLMs Capture Developers' Practical Needs for Code Summarization? An Empirical Study of Code Summary Optimization Behaviors on GitHub
Xianwei Wu, Haifeng Shen, Guoping Rong
Internetware 2026

Motivation

Existing code summarization datasets mainly fall into two paradigms:

• Static generation: one-to-one code–summary mapping (e.g., CodeSearchNet)
• Passive synchronization: summary updates triggered by code changes (e.g., CUP)

In practice, developers frequently perform proactive summary optimization—correcting facts, adding missing information, improving readability, or aligning with documentation standards—while leaving the code unchanged. CSO-Bench is built from these docstring-only commits to evaluate LLMs on this overlooked setting.

Dataset Overview

Split	Granularity	Format	Size	Notes
Train / Knowledge Base	Method	JSONL	3,645	Supports RAG retrieval
Train / Knowledge Base	Class	JSONL	800	Supports RAG retrieval
Test / Evaluation	Method	JSON	200	50 per quality dimension
Test / Evaluation	Class	JSON	200	50 per quality dimension
Total			4,845	Derived from 13,736 docstring-only commits across 22 Python projects

Quality Dimensions

Each evaluation instance is labeled with one primary optimization intent:

Dimension	Description
Factuality	Correcting incorrect information in the summary
Completeness	Supplementing missing but necessary information
Clarity	Improving readability and conciseness
Compliance	Aligning with documentation standards / conventions

Tasks

CSO-Bench supports two complementary evaluation tasks:

1. Summary Judgment
   Given code context and a randomly ordered pair of summaries {S_old, S_ref}, the model selects which version better meets developers' needs.

2. Summary Editing
   Given code context and the defective summary S_old, the model generates an improved summary S_gen.

Repository Structure

CSO-Bench/
├── Data/
│   ├── Train/
│   │   ├── train_method.json   # JSONL, 3,645 method-level samples
│   │   └── train_class.json    # JSONL, 800 class-level samples
│   └── Test/
│       ├── test_method.json    # JSON, keyed by quality dimension
│       └── test_class.json     # JSON, keyed by quality dimension
├── Process/
│   └── parse_enhanced_code_doc.py  # Commit mining / docstring-change parsing
└── README.md

Data Format

Training files (Data/Train/*.json)

JSON Lines (one instance per line).

Test files (Data/Test/*.json)

A JSON object keyed by quality dimension:

{
  "Factuality": [ ... ],
  "Completeness": [ ... ],
  "Clarity": [ ... ],
  "Compliance": [ ... ]
}

Instance schema

Each instance contains:

Field	Description
id	Unique sample identifier
modify_file	Relative path of the modified source file
modify_item	Method / class identifier
origin_version	Pre-change code and docstring
new_version	Post-change code and docstring (developer-refined reference)
meta_data	Provenance metadata (project, commit SHA, message, timestamp)

origin_version / new_version fields:

Field	Description
language	Programming language (python)
identifier	Method / class name
parameters	Parameter list
argument_list	Argument / inheritance list when applicable
return_statement	Return statement snippet when available
docstring	Full docstring text
function	Function source (method-level)
class	Class source (class-level)

Example (truncated):

{
  "id": "...",
  "modify_file": "sklearn/metrics/pairwise.py",
  "modify_item": "pairwise_distances",
  "origin_version": {
    "language": "python",
    "identifier": "pairwise_distances",
    "docstring": "...",
    "function": "def pairwise_distances(...):\n    ..."
  },
  "new_version": {
    "language": "python",
    "identifier": "pairwise_distances",
    "docstring": "...",
    "function": "def pairwise_distances(...):\n    ..."
  },
  "meta_data": {
    "commit_id": "...",
    "project": "scikit-learn/scikit-learn",
    "language": "python",
    "commit_context": {
      "commit_sha": "...",
      "commit_date_time": "...",
      "commit_message": "..."
    }
  }
}

Quick Start

Load the evaluation set

import json
from pathlib import Path

def load_test(path: str):
    with open(path, "r", encoding="utf-8") as f:
        data = json.load(f)
    instances = []
    for dimension, items in data.items():
        for item in items:
            item = dict(item)
            item["dimension"] = dimension
            instances.append(item)
    return instances

method_test = load_test("Data/Test/test_method.json")
class_test = load_test("Data/Test/test_class.json")
print(len(method_test), len(class_test))  # 200 200

Load the training / knowledge-base set

import json

def load_jsonl(path: str):
    with open(path, "r", encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]

method_train = load_jsonl("Data/Train/train_method.json")
class_train = load_jsonl("Data/Train/train_class.json")
print(len(method_train), len(class_train))  # 3645 800

Minimal task views

# Summary Judgment: choose the better summary between old and new
code = item["new_version"]["function"] or item["new_version"]["class"]
s_old = item["origin_version"]["docstring"]
s_ref = item["new_version"]["docstring"]

# Summary Editing: improve the old summary given code
s_gen = model.edit(code=code, summary=s_old)

Construction Pipeline

At a high level, CSO-Bench is constructed as follows:

1. Repository selection from popular open-source Python projects with permissive licenses
2. Commit parsing to identify docstring-only updates at method / class granularity (Process/parse_enhanced_code_doc.py)
3. Multi-stage filtering (context-driven updates, low-information removal, tiny-change filtering, LLM filtering, deduplication)
4. Thematic analysis to define quality dimensions
5. Manual annotation to obtain a balanced high-quality evaluation set (50 samples × 4 dimensions × 2 granularities)

Note: Process/parse_enhanced_code_doc.py demonstrates the commit-level parsing logic used to mine code/docstring changes. Running it end-to-end may require additional local dependencies (e.g., Tree-sitter-based parsers, PyDriller) and project lists.

Citation

If you use CSO-Bench in your research, please cite:

@inproceedings{wu2026csobench,
  title     = {Can LLMs Capture Developers' Practical Needs for Code Summarization? An Empirical Study of Code Summary Optimization Behaviors on GitHub},
  author    = {Wu, Xianwei and Shen, Haifeng and Rong, Guoping},
  booktitle = {Proceedings of the 17th International Conference on Internetware (Internetware 2026)},
  year      = {2026},
  address   = {Gold Coast, QLD, Australia}
}

License

The dataset is constructed from publicly available open-source repositories. Please respect the original licenses of the source projects when redistributing derived artifacts. If a project-level license file is added to this repository, it supersedes this note.

Contact

For questions or issues related to CSO-Bench, please open a GitHub issue or contact the authors of the paper.