humaneval

We introduced a new model designed for the Code generation task. Its test accuracy on the HumanEval base dataset surpasses that of GPT-4 Turbo (April 2024) and GPT-4o.

Self-evaluating interview for AI coders

Run evaluation on LLMs using human-eval benchmark

SkyCode是一个多语言开源编程大模型，采用GPT3模型结构，支持Java, JavaScript, C, C++, Python, Go, shell等多种主流编程语言，并能理解中文注释。模型可以对代码进行补全，拥有强大解题能力，使您从编程中解放出来，专心于解决更重要的问题。| SkyCode is an open source programming model, which adopts the GPT3 model structure. It supports Java, JavaScript, C, C++, Python, Go, shell and other languages, and can understand Chinese comments.

Evaluate LLM-generated COBOL

Run Claude Code with any foundation model on Amazon Bedrock (43 models) or a self-hosted model on EC2. Includes a HumanEval benchmark.

A strict, auditable HumanEval benchmark runner for GGUF models served via llama.cpp, using its OpenAI-compatible HTTP API.

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

Specification for a reproducible, provenance-bound multi-lane LLM benchmark suite.

A pip-installable benchmark runner for LLMs and agents. Five minutes to your first eval.

HumanEval 模型评估工具

Prompt-engineering study on LLM math reasoning (GSM8K) and code generation (HumanEval): zero/few-shot, self-consistency, self-verification, and experiments on prompt quality, complexity, demonstrations, and diversity.

Behavioral Trust Clustering a thermodynamic governance layer that reduces LLM hallucination by 52% on HumanEval. Drop-in wrapper for any decoder. MIT.

Benchmark suite for evaluating LLMs and SLMs on coding and SE tasks. Features HumanEval, MBPP, SWE-bench, and BigCodeBench with an interactive Streamlit UI. Supports cloud APIs (OpenAI, Anthropic, Google) and local models via Ollama. Tracks pass rates, latency, token usage, and costs.

LLM Collaboration for Code Generation

A lightweight, configuration-driven evaluation framework for LLM code generation & reasoning tasks (MBPP, HumanEval, GSM8K). Supports multi-provider (DeepSeek, OpenAI, ZhipuAI) and concurrent execution.

Lightweight LLM evaluation tool for OpenAI-compatible endpoints (Chat Completions / Responses) with CLI and local Web UI. MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag — bring your own BASE_URL + MODEL_NAME.

Experimental self-verifying AI agent — generates multiple candidate solutions, tests them in an isolated Docker sandbox, and verifies, selects, and repairs them using real execution evidence rather than model self-assessment. Evaluated on HumanEval: 92% → 100% task success.

Self-improvement research testbed for LLMs: Dream-RSI-style policy evolution (arXiv:2609.14858) + STaR-style weight upgrades, with a Rust rootless sandbox. Apache-2.0.

Orquestación de equipos de agentes LLM con roles especializados para resolución colaborativa de tareas complejas de ingeniería de software