LiveCodeBench

LiveCodeBench (LCB) is a continuously versioned benchmark and open-source evaluation framework for measuring the programming capabilities of large language models (LLMs). It was introduced by Naman Jain and colleagues in the 2024 paper LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, which was subsequently published at the International Conference on Learning Representations in 2025.

The benchmark collects recently released competitive-programming problems from LeetCode, AtCoder, and Codeforces. Every problem is associated with its contest date, allowing evaluators to restrict a test to questions published after a model's reported training cutoff or release date. This time-segmented evaluation is intended to reduce the influence of benchmark contamination and memorization that can affect older, static coding datasets.

The original design evaluates four distinct capabilities:

• Code generation from a natural-language problem statement.
• Self-repair after receiving execution or test feedback.
• Code execution, in which the model predicts the result of running a program.
• Test output prediction, in which the model infers the expected output of a specified test input without being given an implementation.

As of 10 July 2026, the official repository identifies release_v6 as the latest public cumulative code-generation release. It contains 1,055 problems published between May 2023 and April 2025. The repository uses a reduced code_generation_lite collection as its default code-generation dataset and currently accepts public leaderboard submissions only for the code-generation scenario.

Background and Motivation

Early code-generation benchmarks such as HumanEval, MBPP, and APPS made it possible to compare language models by executing generated programs against tests. Over time, however, several limitations became increasingly important:

• The benchmark problems and reference solutions were publicly available and could enter pre-training or instruction-tuning corpora.
• Small and fixed datasets were vulnerable to saturation and benchmark-specific fine-tuning.
• Some datasets provided too few tests to reliably detect incorrect implementations.
• Competitive-programming collections could include interactive tasks or problems accepting multiple valid outputs, making ordinary exact-output grading unreliable.
• Most established benchmarks measured only natural-language-to-code generation, even though practical programming systems also need debugging, execution tracing, and test reasoning.

LiveCodeBench was designed to address these problems through repeated collection of newly published contest tasks, date-based filtering, stronger executable tests, and several evaluation scenarios. Its authors described the benchmark as “contamination free”; in practice, the methodology is more accurately understood as contamination-aware. It reduces exposure risk by evaluating models on post-cutoff problems but cannot prove that every evaluated item was absent from every stage of model development.

The benchmark also attempts to preserve meaningful difficulty variation. If every included problem is too easy, leading models converge near 100% accuracy. If every problem is at advanced Olympiad level, most models score close to zero. LiveCodeBench therefore maps source-platform ratings into the broad categories Easy, Medium, and Hard and reports performance for the categories separately.

Design Principles

Principle	Implementation	Intended benefit
Temporal freshness	New contest problems are periodically added and tagged with their publication dates.	Allows evaluation on problems unlikely to have appeared in older model-training data.
Time-window filtering	Scores can be calculated between specified start and end dates.	Makes it possible to align an evaluation with a model's stated training cutoff or public release.
Holistic evaluation	The original framework contains code generation, self-repair, code execution, and test output prediction.	Separates capabilities that are often combined inside programming agents.
Executable grading	Generated programs and predicted outputs are checked through execution or exact comparison.	Reduces dependence on human annotators and subjective LLM judges.
Problem curation	Interactive, image-dependent, ambiguous, and non-autogradable problems are filtered where possible.	Reduces false failures caused by unsuitable task formats.
Robust tests	Platform tests are supplemented with generated and adversarial inputs when necessary.	Detects programs that solve only visible examples or common cases.
Difficulty diversity	Tasks are normalized into Easy, Medium, and Hard groups.	Supports both broad model comparison and capability analysis by difficulty.
Versioned releases	Cumulative and incremental version tags identify the exact problem set used.	Improves reproducibility and permits historical comparisons.

Data Collection and Curation

Source Platforms

LiveCodeBench obtains problems from three public competitive-programming platforms.

Platform	Original collection policy	Tests and formatting
LeetCode	Weekly and biweekly contest problems published after April 2023 were collected. The platform's Easy, Medium, and Hard labels were retained.	Starter code and public examples were collected. Because complete hidden tests are not publicly exposed, the benchmark generated additional inputs and incorporated selected failing tests returned by the platform grader.
AtCoder	The original study concentrated on AtCoder Beginner Contests and excluded the more advanced Regular and Grand Contests. Problems above the project's selected difficulty threshold were omitted.	AtCoder's public and hidden tests were used directly when available. Numeric platform ratings were mapped into the three LiveCodeBench difficulty groups.
Codeforces	The initial collection used a limited number of Division 3 and Division 4 problems because even these were generally more difficult than the selected LeetCode and AtCoder tasks.	Because complete long tests are not always exposed on the website, test generators were constructed semi-automatically.

Automated HTML scrapers collect the problem statement and metadata. Mathematical expressions are parsed into a normalized representation, while tasks that depend on images are excluded. The collection process also attempts to obtain public tests, hidden tests where accessible, accepted human solutions, source-platform difficulty ratings, contest identifiers, and publication dates.

Problem Filtering

Not every competition problem is suitable for deterministic input-output evaluation. The original filtering process excluded or attempted to detect:

• Interactive tasks that require repeated communication with a judge.
• Problems allowing several structurally different valid outputs for the same input.
• Construction tasks for which a single reference output is not sufficient.
• Problems whose essential information is contained in an image.
• Tasks that cannot be reproduced reliably outside the source platform.
• Problems beyond the benchmark's selected difficulty range.

The filtering system used keyword-based heuristics and source-specific rules. These procedures made continuous collection practical, but they were not infallible. The maintainers therefore publish an errata file documenting tasks with erroneous tests, multiple acceptable outputs, unsupported interaction patterns, or other grading problems.

Test Construction

Tests are central to every LiveCodeBench scenario. When official tests are available, they are used directly. Otherwise, the original pipeline employed a language model to write input-generator programs based on the problem specification. Those generators produced diverse valid test inputs rather than asking the language model to list a small number of test cases directly.

The collection may combine:

• Public sample tests from the problem statement.
• Official hidden tests supplied by a contest platform.
• Inputs sampled from generated test-case generators.
• Adversarial or failing tests obtained from platform graders.
• Boundary cases derived from constraints in the problem statement.

In the May 2023–May 2024 snapshot analyzed in the main paper, the code-generation collection contained an average of approximately 17 tests per problem. The authors found that measured performance generally stabilized after approximately 20 private inputs, providing a trade-off between grading cost and test coverage.

Paper-Era Dataset Composition

The main ICLR study analyzed a cumulative code-generation snapshot containing 511 problems released between May 2023 and May 2024.

Composition of the May 2023–May 2024 code-generation snapshot

Source or split	Problems	Easy	Medium	Hard	Average tests
Complete snapshot	511	182	206	123	17.0
AtCoder	267	99	91	77	15.6
LeetCode	235	79	113	43	19.0
Codeforces	9	4	2	3	11.1

These values describe the historical paper snapshot rather than the later release_v6 collection. The platform proportions, difficulty distribution, and number of tests can change as new cumulative releases are added.

Evaluation Scenarios

Code Generation

In the code-generation scenario, the model receives:

• A natural-language problem statement.
• Input and output specifications.
• Constraints.
• Public input-output examples.
• Starter code where the source task uses a function-based interface.

The model must generate a complete program or function. Its output is executed against private tests, and a problem is counted as solved only if the program produces the expected result for every evaluated test within the time and memory limits.

LeetCode tasks commonly use a function-call format with provided starter code. AtCoder and Codeforces tasks normally use the standard-input and standard-output format customary in programming contests.

Self-Repair

The self-repair scenario begins with a program generated during the code-generation evaluation. If the program is incorrect, the model receives feedback describing the failure and is asked to produce a corrected implementation.

Feedback can include:

• A syntax error.
• A runtime exception.
• A failing test input.
• The program's output and the expected output.
• A time-limit error.

The final repaired program is executed against the benchmark tests. The original metric measures whether either the initial implementation or the one-step repaired implementation passes the tests. As a result, a self-repair score reflects both initial generation ability and the ability to use feedback; it is not a pure measure of debugging in isolation.

Code Execution

The code-execution scenario evaluates whether a model can mentally simulate a program. It provides:

• A Python function.
• A concrete function call or assertion.
• A placeholder for the resulting value.

The model must predict the literal output produced by running the code. The answer is checked through execution and exact comparison. The original experiments evaluated both direct prediction and a chain-of-thought configuration in which the model was instructed to trace the program step by step.

Code execution tests comprehension of control flow, state mutation, recursion, data structures, and language semantics without requiring the model to design a new algorithm.

Test Output Prediction

In test output prediction, the model receives a natural-language problem specification and a fixed test input but not a reference implementation. It must infer the output that a correct implementation should produce.

This scenario differs from ordinary test generation. The benchmark supplies the input, so the evaluator only needs to check the predicted output. The task measures whether the model understands the specification well enough to reason through a concrete case.

Scenario	Input to the model	Required output	Main capability	Grading
Code generation	Problem statement and examples	Executable solution	Algorithm design and implementation	Hidden-test functional correctness
Self-repair	Problem, incorrect solution, and failure feedback	Corrected solution	Debugging and feedback use	Final-program functional correctness
Code execution	Program and concrete input	Actual runtime output	Program simulation and code comprehension	Exact execution-based comparison
Test output prediction	Problem statement and fixed input	Expected output of a correct solution	Specification understanding and concrete reasoning	Exact comparison

Dataset Versions

LiveCodeBench uses cumulative release tags. Every release_vN contains the earlier problems plus tasks added in subsequent collection periods.

Release	Publication interval represented	Cumulative problem count
release_v1	May 2023 – March 2024	400
release_v2	May 2023 – May 2024	511
release_v3	May 2023 – July 2024	612
release_v4	May 2023 – September 2024	713
release_v5	May 2023 – January 2025	880
release_v6	May 2023 – April 2025	1,055

The runner defaults to release_latest, which resolves to the latest version recognized by the installed code. An evaluator should nevertheless record the explicit release tag because the meaning of release_latest can change after a repository update.

The project also defines fine-grained tags such as:

• v1 — problems added in the first collection period.
• v2 — problems newly added for the second release.
• v1_v3 — a union of selected incremental periods.
• v4_v5 — problems belonging to the fourth and fifth update intervals.

This distinction is important: release_v5 is cumulative, while v5 refers only to the problems introduced in that update.

As of 10 July 2026, no cumulative version later than release_v6 is listed in the official repository. The newest problems in the public cumulative set were therefore published in April 2025.

Data Format and Software

Code-Generation Records

The public code_generation_lite dataset commonly exposes the following fields:

Field	Description
question_title	Human-readable title of the programming problem.
question_content	Full problem statement, including constraints and input-output descriptions.
platform	Source platform, such as LeetCode, AtCoder, or Codeforces.
question_id	Benchmark or source-specific problem identifier.
contest_id	Identifier of the source contest.
contest_date	Publication date used for time-window filtering.
starter_code	Function signature or code template where applicable.
difficulty	Normalized Easy, Medium, or Hard label.
public_test_cases	Examples visible to the model or included in the problem statement.
private_test_cases	Hidden tests used for executable grading.
metadata	Additional information needed to parse, execute, or grade the task.

Some fields contain serialized JSON structures rather than simple strings. Consumers should use the official loader or runner instead of assuming that all test data can be interpreted through direct string splitting.

A simplified conceptual record is:

{
  "question_title": "Example Contest Problem",
  "question_content": "Full natural-language specification",
  "platform": "atcoder",
  "question_id": "example_id",
  "contest_id": "example_contest",
  "contest_date": "2025-01-01",
  "starter_code": "",
  "difficulty": "medium",
  "public_test_cases": "serialized public tests",
  "private_test_cases": "serialized private tests",
  "metadata": "serialized grading metadata"
}

Code Generation Lite

The original code-generation dataset contained a large number of test cases and required considerable storage and evaluation time. The maintainers later produced code_generation_lite by pruning and sampling tests while attempting to preserve similar aggregate model rankings and performance estimates. The lite collection became the runner's default; the full version can be requested with a non-fast evaluation option.

The lite version improves evaluation speed but creates a trade-off: a smaller test suite has a greater chance of failing to expose an unusual incorrect implementation.

Official Runner

The official repository contains model-specific prompt templates, response parsers, executable checkers, score aggregation tools, and support for local or API-hosted models. A typical code-generation command is:

python -m lcb_runner.runner.main \
  --model MODEL_NAME \
  --scenario codegeneration \
  --evaluate \
  --release_version release_v6

Scores for a specific publication window can be calculated separately:

python -m lcb_runner.evaluation.compute_scores \
  --eval_all_file PATH_TO_RESULTS \
  --start_date 2024-01-01 \
  --end_date 2025-01-31

The evaluation software is released under the MIT License. The contest problem statements, tests, and accepted solutions originate from external platforms; the original paper states that publicly visible material was collected for academic evaluation under a fair-use rationale. The legal status and redistribution terms of the source content are distinct from the software license.

Evaluation Methodology

Functional Correctness

A generated program succeeds only if it:

• Can be extracted from the model response.
• Compiles or executes in the target environment.
• Finishes within the allowed time.
• Produces the accepted output for every selected private test.

A response can therefore fail because of an algorithmic error, implementation bug, output-format mismatch, unsupported dependency, timeout, memory issue, or failure of the response parser to recover the intended code.

The principal metric is pass@1. It estimates the probability that one sampled model completion solves a problem. The repository also computes pass@5, which estimates the probability that at least one of five sampled completions is correct.

In plain form:

• pass@1 — expected success rate for one generated solution per problem.
• pass@5 — expected success rate when up to five generated solutions may be tried.

Pass@5 is normally higher than pass@1 but uses a larger inference budget. The two metrics should not be compared as though they represented the same deployment setting.

Sampling Configuration

The original ICLR evaluation generated ten candidate responses per problem using nucleus sampling with:

• Temperature: 0.2.
• Top-p: 0.95.
• Ten sampled candidates.

Instruction-tuned models received a zero-shot code-generation prompt, while base models used a one-shot prompt to reduce formatting failures.

The official repository's documented defaults in July 2026 also use ten generations and a temperature of 0.2, and its code-generation evaluator reports pass@1 and pass@5.

Alternative reports may instead use greedy decoding, one completion, much larger sampling budgets, majority voting, execution-guided selection, or agentic repair. Such results measure different combinations of base-model capability and test-time computation.

Time-Segmented Evaluation

Every problem has a contest date. The evaluator can select only tasks whose release dates fall after a model's reported training cutoff.

For example, a model with a stated cutoff of 31 December 2024 could be evaluated on problems beginning 1 January 2025. In principle, this reduces the likelihood that exact questions or official solutions appeared in pre-training.

The procedure has several caveats:

• A published cutoff may describe only one stage of model training.
• Post-training, reinforcement learning, retrieval indexes, or evaluation-specific development may use newer data.
• Model providers do not always disclose a precise cutoff.
• A model's release date is only a rough substitute when its cutoff is unknown.
• Contest problems may circulate privately or appear in derived datasets before the recorded public date.

Time filtering therefore provides evidence against direct contamination but not a mathematical guarantee of data independence.

Difficulty, Platform, and Time Reporting

A single aggregate score can obscure important differences. LiveCodeBench supports breakdowns by:

• Easy, Medium, and Hard difficulty.
• Source platform.
• Monthly or custom date window.
• Cumulative or incremental release.
• Evaluation scenario.

Platform-specific results are useful because task style and test construction differ across LeetCode, AtCoder, and Codeforces. A model may perform differently on function-completion problems than on standard-input contest programs even when the underlying algorithmic difficulty is similar.

Runtime Variability

The official repository notes that time limits and evaluation concurrency can cause small variations of less than approximately 0.5 score points. Maintainers recommend reducing the number of parallel evaluation processes or increasing the timeout if larger inconsistencies appear.

This variability particularly affects solutions close to the time limit, programs with unstable resource consumption, and shared execution environments.

Reproducibility Requirements

A complete LiveCodeBench result should identify:

• Exact model and checkpoint or API version.
• Dataset release tag.
• Start and end dates.
• Programming language.
• Scenario.
• Prompt and chat template.
• Number of generated samples.
• Temperature, top-p, and decoding method.
• Maximum output length.
• Tool and execution access.
• Whether reasoning text was visible or hidden.
• Code-extraction procedure.
• Full or lite test collection.
• Pass@1, pass@5, or another aggregation metric.
• Timeout and execution environment.
• Any repair, reranking, voting, or verifier stage.

A score labelled only “LiveCodeBench” is ambiguous because it may refer to a different release, time window, language, sampling budget, or evaluation pipeline.

Original Study and Findings

The ICLR study evaluated 52 models ranging from 1.3 billion to 70 billion parameters, including base models, instruction-tuned open models, and proprietary API systems. To reduce likely contamination in the DeepSeek family, its main comparative results used problems released from September 2023 onward.

Historical Code-Generation Results

Selected code-generation results from the paper are shown below. They use the post-August-2023 time window, ten sampled responses per problem, temperature 0.2, and top-p 0.95.

Selected results from the ICLR study

Model	Easy pass@1	Medium pass@1	Hard pass@1	Overall pass@1
GPT-4o, 13 May 2024	88.3%	33.2%	4.2%	41.9%
GPT-4 Turbo, 9 April 2024	85.3%	33.0%	5.1%	41.1%
Claude 3 Opus	78.8%	16.3%	3.2%	32.8%
Llama 3 70B Instruct	60.7%	15.8%	1.4%	26.0%

These figures are historical results from a 2024-era model set, not a current leaderboard. They nevertheless illustrate the sharp difficulty gradient: even the strongest evaluated systems solved most Easy problems but only a small fraction of Hard problems.

Evidence of Contamination

The researchers plotted model performance by problem publication month. Several systems showed abrupt declines around their reported training cutoffs or release dates, especially on LeetCode tasks.

Examples discussed in the paper included:

• DeepSeek-Coder models declining on LeetCode problems published after August 2023.
• GPT-4o declining on problems published after its reported late-2023 cutoff.
• More recent systems showing similar boundaries around their release periods.
• Considerably smoother month-to-month performance on AtCoder for the same models.

The authors interpreted these patterns as evidence that higher scores on older LeetCode questions could be partly explained by training exposure. The result also demonstrated why one fixed aggregate score can conceal temporal contamination.

A temporal decline is not definitive proof of memorization. Newer tasks may differ in style or difficulty, and a source platform may change its contest distribution over time. The evidence is strongest when a discontinuity aligns with an independently reported cutoff and is not reproduced on other platforms.

Holistic Capability Differences

Scores across the four scenarios were strongly correlated, with pairwise correlations above 0.88 in the original study. Strong code-generation models generally also performed well on repair, execution, and output prediction.

The relative gaps were not constant, however. The paper found that:

• GPT-4 Turbo gained a larger advantage in self-repair than in direct generation.
• Claude 3 Opus and Mistral Large performed comparatively well on tasks involving execution reasoning.
• Claude 3 Opus exceeded GPT-4 Turbo on test output prediction in the evaluated setup.
• Chain-of-thought substantially improved code execution for several proprietary models but was less consistently beneficial for open models.

These differences support the benchmark's original argument that one code-generation score does not fully characterize a programming model.

Comparison with HumanEval

The authors compared pass@1 on HumanEval+ with performance on the Easy subset of LiveCodeBench. The reported correlation was only 0.72, and several fine-tuned open models performed much better on HumanEval+ than on recent contest problems.

For example, DeepSeek-Coder-Instruct 1.3B achieved approximately 60% on HumanEval+ but only 26% on LCB-Easy in the paper's evaluation. Other fine-tuned systems displayed similar gaps.

The authors described this as evidence of possible overfitting to HumanEval or closely related training distributions. The comparison does not establish that any particular model was trained directly on HumanEval, but it shows that strong performance on its short function-completion tasks did not always transfer to more varied contest problems.

Adoption and Current Status

LiveCodeBench became a frequently reported benchmark in code-model technical reports, model cards, reasoning-model evaluations, reinforcement-learning studies, and agentic code-generation research. Its principal attractions include:

• Public executable tests.
• Straightforward pass@k metrics.
• Explicit problem dates.
• Several difficulty levels.
• More challenging tasks than small function-synthesis benchmarks.
• A reusable local evaluation runner.
• Support for arbitrary temporal slices.

The project's public identity has nevertheless shifted toward code generation. The repository still contains commands for self-repair, test output prediction, and code execution, but its July 2026 submission instructions accept new leaderboard entries only for code generation.

The public release cadence is another important status distinction. Although LiveCodeBench was designed as a continuously updated benchmark, the latest cumulative public version listed on 10 July 2026 ends in April 2025. The “live” property therefore depends on continued collection and release activity rather than being an automatic property of the evaluation software.

Related Extensions

LiveCodeBench Pro

LiveCodeBench Pro is a separate benchmark introduced in 2025 to evaluate substantially more difficult competitive-programming tasks. It draws problems from Codeforces, the International Collegiate Programming Contest (ICPC), and the International Olympiad in Informatics (IOI).

A team of international programming-contest medalists annotated the problems by algorithmic category and performed line-by-line analysis of failed model submissions. In the original study's no-tool setting, the strongest evaluated model achieved 53% pass@1 on Medium problems and 0% on Hard problems.

The researchers reported that models were comparatively strong on implementation-heavy tasks but continued to struggle with:

• Identifying subtle algorithmic invariants.
• Complex case analysis.
• Proving that a proposed algorithm is correct.
• Distinguishing a plausible heuristic from a complete solution.
• Detecting confidently stated but invalid reasoning.

LiveCodeBench Pro is not release_v7 and should not be treated as a cumulative update of the standard LiveCodeBench dataset. It has a different problem distribution, curation process, and difficulty target.

Multi-LCB

Multi-LCB is a multilingual programming-language extension introduced in a June 2026 preprint. It converts LiveCodeBench code-generation tasks into equivalent evaluation problems for 12 programming languages, including Python.

The supported language set includes:

• Python.
• C++.
• C#.
• Java.
• Rust.
• Go.
• TypeScript.
• JavaScript.
• Ruby.
• PHP.
• Kotlin.
• Scala.

Multi-LCB preserves the original task semantics, temporal metadata, and executable evaluation principle while converting function-oriented problems into a more uniform standard-input and standard-output representation. Its initial study evaluated 24 language models and reported evidence of Python-specific overfitting, language-dependent contamination, and substantial cross-language performance differences.

Multi-LCB addresses the original benchmark's principal language limitation but is an independent extension rather than an official cumulative LiveCodeBench release.

Benchmark	Introduced	Primary focus	Relationship to LiveCodeBench
LiveCodeBench	2024	Recent Easy-to-Hard contest problems and four coding scenarios	Original benchmark program
LiveCodeBench Pro	2025	Expert-level Codeforces, ICPC, and IOI problems	Separate harder benchmark with specialist annotations
Multi-LCB	2026	Code generation across 12 programming languages	Independent multilingual-language extension

Limitations and Criticism

Conditional Contamination Control

The term “contamination free” can be misleading when interpreted as an absolute property. LiveCodeBench does not inspect every token used during pre-training, post-training, reinforcement learning, retrieval indexing, model evaluation, or prompt development.

Time segmentation assumes that a problem published after a reported cutoff was unavailable to the model. This assumption can fail when:

• The stated cutoff applies only to pre-training.
• A later instruction-tuning corpus includes recent contest problems.
• A model has access to external search or retrieval.
• The problem was shared before its official contest date.
• A derived or translated version entered training data.
• The provider does not disclose the relevant development timeline.

The benchmark is therefore best described as reducing the probability of contamination under documented evaluation conditions.

Public Data Eventually Becomes Training Data

After a LiveCodeBench release is published, its problem statements, tests, metadata, prompts, and evaluation code become available to model developers. Public model repositories already identify systems trained or fine-tuned on the code-generation dataset.

A model can consequently be optimized directly for older LiveCodeBench releases. Temporal filtering remains useful only when sufficiently new holdout problems continue to be added.

Update Cadence

As of July 2026, release_v6 remains the latest cumulative version and contains no problems newer than April 2025. This creates a growing gap between the newest public benchmark items and recently developed models.

A versioned but no longer frequently updated benchmark retains value for reproducibility, but it gradually loses the contamination resistance implied by the word “Live”.

Shrinking Post-Cutoff Samples

Evaluating only post-cutoff problems reduces the available sample. In the original paper, filtering the 511-problem collection to the post-August-2023 interval left 349 code-generation problems. Bootstrap analysis suggested approximately one to two percentage points of variation for samples of that size.

The problem becomes more severe for newer models. A late cutoff may leave only a small number of eligible questions, particularly after dividing the data by platform and difficulty.

Small differences between leaderboard scores may therefore be statistically uninformative unless confidence intervals or repeated samples are reported.

Cross-Version Comparability

Scores from release_v1, release_v4, and release_v6 are not directly interchangeable. Later releases contain more tasks and can have different proportions of:

• Source platforms.
• Difficulty levels.
• Function-based and standard-input formats.
• Algorithmic topics.
• Test-suite sizes.
• Publication periods.

Two reports using the label “LiveCodeBench v6” may also select different date windows. Exact release and filtering parameters are necessary for meaningful comparison.

Test and Grading Errors

Executable tests provide stronger evidence than natural-language judgment, but no finite test suite proves that a program is correct for every valid input.

Potential problems include:

• Missing boundary cases.
• Incorrect expected outputs.
• Generated invalid inputs.
• Multiple valid outputs not recognized by the checker.
• Floating-point or whitespace normalization errors.
• Platform-specific behavior.
• Timeout instability.
• A correct but unusually slow solution being rejected.
• An incorrect solution passing a reduced lite test set.

The official errata demonstrates that some unsuitable problems and erroneous tests passed the automated curation process. Results can change when affected items or tests are corrected.

Lite Test-Suite Trade-Off

The code_generation_lite collection reduces evaluation cost by removing or sampling tests. The maintainers report that it produces performance estimates similar to the original collection, but aggregate similarity does not guarantee identical decisions for every model or problem.

A generated program can pass the lite set and fail a removed test. Comparisons should not mix full and lite evaluations without identifying the difference.

Competitive Programming Is Not General Software Engineering

LiveCodeBench measures the ability to solve self-contained algorithmic problems. This is only one part of programming.

The benchmark does not comprehensively evaluate:

• Navigating a large existing repository.
• Understanding undocumented architectural conventions.
• Modifying several interacting files.
• Using unfamiliar third-party APIs.
• Writing database migrations.
• Debugging distributed systems.
• Preserving backward compatibility.
• Reviewing code written by another developer.
• Handling security, privacy, and reliability requirements.
• Communicating with users and maintainers.
• Long-term maintenance of a software project.

A system can obtain a high LiveCodeBench score while remaining weak on repository-level software-engineering tasks. Benchmarks such as SWE-bench target a different and complementary capability.

Python-Centered Design

The original benchmark's main tasks, prompts, tooling, and analysis were centered on Python, although the ICLR paper included a limited supplementary C++ comparison.

Python performance does not automatically transfer to languages with:

• Manual memory management.
• Static type systems.
• Ownership and borrowing rules.
• Complex compilation models.
• Different standard libraries.
• Concurrency-specific semantics.
• More verbose input-output requirements.

Multi-LCB was introduced specifically to provide systematic coverage of additional programming languages.

Metric Blind Spots

Pass@k evaluates observable functional correctness under the provided tests. It does not directly reward:

• Readability.
• Maintainability.
• Modularity.
• Documentation.
• Security.
• Numerical robustness beyond tested inputs.
• Explanatory quality.
• Algorithmic originality.
• Faithfulness of the model's reasoning.
• Energy or inference cost.
• Compliance with project-specific style rules.

A program containing incorrect reasoning can receive full credit if the final implementation passes. Conversely, a sound solution may fail because of a minor output-format or extraction error.

Sampling and Test-Time Compute

Pass@1 from one greedy output is not equivalent to pass@1 estimated from ten stochastic samples. Pass@5, majority voting, verifier reranking, execution-guided repair, and multi-agent workflows use progressively larger computational budgets.

Modern reasoning models may also generate very long internal traces or repeatedly execute candidate programs. A higher score can reflect:

• A stronger base model.
• More sampled candidates.
• A larger reasoning-token allowance.
• Better test generation.
• An external verifier.
• More repair iterations.
• Tool-assisted search.

Reports should distinguish model capability from the evaluation-time system surrounding the model.

Prompt and Parser Sensitivity

Model outputs are affected by system prompts, chat templates, code fences, language instructions, and answer-extraction rules. A model may generate a correct program surrounded by explanatory text that a parser fails to extract. Another parser may recover the same code successfully.

The original paper used mostly standardized prompts with limited model-family adjustments, but it acknowledged that prompt choice could change results.

Self-Repair Metric Interpretation

The original self-repair score counts success after either the first solution or the repair attempt. A model with strong initial generation and weak debugging can therefore outperform a better debugger whose starting solutions are poorer.

A dedicated repair analysis should report at least:

• Initial code-generation accuracy.
• Number of initially failed programs eligible for repair.
• Fraction of failed programs repaired successfully.
• Error category.
• Number of repair iterations.
• Information included in the feedback.

Source Content and Redistribution

The evaluation software has an explicit open-source license, but the underlying problems originate from external contest platforms. Problem statements, editorials, accepted solutions, and hidden tests can have separate terms of use or copyright ownership.

Researchers redistributing modified datasets or commercial evaluation services should distinguish the license of the runner from the legal status of source-platform content.

External links

• Official LiveCodeBench website
• Official LiveCodeBench repository
• LiveCodeBench code-generation dataset
• Official leaderboard
• LiveCodeBench paper in the ICLR proceedings
• LiveCodeBench Pro paper
• Multi-LCB paper

See also

• Large language model evaluation
• HumanEval
• SWE-bench

Literature

• Jain, N. et al. (2025). LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. ICLR 2025. arXiv:2403.07974.
• Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
• Hendrycks, D. et al. (2021). Measuring Coding Challenge Competence With APPS. NeurIPS. arXiv:2105.09938.
• Liu, J. et al. (2023). Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. NeurIPS. arXiv:2305.01210.
• Zheng, Z. et al. (2025). LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?. arXiv:2506.11928.
• Ivanova, M. et al. (2026). Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages. arXiv:2606.20517.