HELM Benchmark

HELM (Holistic Evaluation of Language Models) is an open-source evaluation framework and benchmark family for measuring the capabilities, limitations, efficiency, and societal risks of large language models (LLMs) and other foundation models. It was developed by the Stanford Center for Research on Foundation Models (CRFM) and introduced by Percy Liang and colleagues in the 2022 paper Holistic Evaluation of Language Models, published in the Transactions on Machine Learning Research in 2023.

HELM differs from a conventional benchmark containing a single fixed dataset and one headline metric. It combines a taxonomy of evaluation scenarios, standardized prompting and adaptation procedures, multiple metrics, a unified model interface, public prompts and responses, reproducible software, and versioned web leaderboards. The name may refer to the general evaluation methodology, the Python software framework, or one of several benchmark suites built with the framework.

The original evaluation, later called HELM Classic, tested 30 language models from 12 organizations on 42 scenarios. It consisted of 16 core scenarios evaluated with seven categories of metrics and 26 additional scenarios organized into seven targeted evaluations. The core metrics were accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.

HELM was initially intended to operate as a continuously updated or “living” benchmark. On 1 June 2026, however, the project entered maintenance mode. Its code, documentation, raw results, and existing leaderboards remain publicly available, but Stanford CRFM no longer plans to add new features or new evaluations to the official leaderboards. The software is maintained by volunteers on a best-effort basis.

Terminology and Scope

The term HELM Benchmark is used for several related concepts:

Meaning	Description
Evaluation methodology	A proposal for evaluating language models through broad scenario coverage, multiple metrics, standardized conditions, and explicit recognition of what remains unevaluated.
HELM Classic	The benchmark configuration introduced in the original paper, consisting of 16 core scenarios and 26 targeted scenarios.
Software framework	An extensible Python package that standardizes datasets, model requests, prompt construction, execution, metrics, result storage, and visualization.
Benchmark family	Versioned suites and leaderboards built with the framework, including HELM Lite, HELM Capabilities, HELM Safety, VHELM, HEIM, MedHELM, and AHELM.
Public result repository	Web interfaces containing aggregate scores together with individual prompts, model completions, run metadata, and downloadable raw results.

The original benchmark concentrated on text-based English-language models. Later projects extended the same design principles to multilingual models, healthcare applications, enterprise tasks, vision-language models, text-to-image systems, and audio-language models.

Background and Motivation

Language models are general-purpose systems that can be adapted to many tasks through prompting, in-context learning, fine-tuning, tools, or other methods. A model that performs strongly on one dataset may nevertheless be unreliable, biased, inefficient, poorly calibrated, or ineffective in another domain.

Before HELM, model developers frequently evaluated different systems on different benchmark selections. The resulting reports were difficult to compare because they varied in:

• Datasets.
• Prompt templates.
• Few-shot examples.
• Decoding parameters.
• Output parsers.
• Metrics.
• Model-access conditions.
• Treatment of failures and missing results.

The HELM researchers examined the evaluation coverage of prominent models and estimated that, before their project, an average model had been tested on only 17.9% of the 16 HELM core scenarios. Some major models had no evaluation dataset in common. HELM increased average coverage to 96.0% by applying a common evaluation protocol to the same model set.

The project was motivated by three related problems:

• Sparse coverage. A small number of accuracy benchmarks cannot characterize the broad range of tasks for which general-purpose models may be deployed.
• Metric narrowness. Accuracy alone does not measure calibration, robustness, group disparities, harmful output, or resource consumption.
• Lack of standardization. Scores obtained with different prompts, examples, parsers, and sampling configurations are not necessarily comparable.

HELM also sought to improve evaluation transparency. Many commercial model reports provide aggregate scores without publishing exact prompts or model responses. HELM instead exposes prompt-level outputs so that users can investigate why a metric changed and whether automatic grading reflects the actual behavior of the model.

Principles of Holistic Evaluation

The original paper defines holistic evaluation through three principal elements.

Principle	Meaning	HELM implementation
Broad coverage and recognition of incompleteness	Evaluation should represent many tasks, domains, languages, and risks while explicitly documenting important omissions.	HELM begins with a taxonomy of possible scenarios and metrics before selecting a feasible benchmark subset.
Multi-metric measurement	Each deployment scenario may involve several desirable or undesirable properties, not only task accuracy.	Core scenarios are evaluated with accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency metrics where applicable.
Standardization	Models should be compared under consistent evaluation conditions so that differences are attributable as far as possible to the systems rather than the harness.	HELM standardizes scenario sampling, prompt adaptation, in-context examples, decoding settings, output processing, metrics, and result formats.

Two additional operational practices became central to the project:

• Transparency. Prompts, completions, intermediate statistics, and aggregate results are made inspectable.
• Reproducibility. The framework, configurations, and result files are released so that third parties can rerun or extend evaluations.

The word “holistic” does not mean that HELM claims to measure every capability or risk. The original paper explicitly identifies missing languages, tasks, metrics, model families, and deployment conditions. Recognition of these gaps is part of the methodology rather than an incidental limitation.

Scenario Taxonomy

HELM uses a scenario as its principal unit of evaluation. In the original conceptual taxonomy, a scenario is characterized by three components:

• Task: what the model is asked to do, such as question answering, summarization, information retrieval, or classification.
• Domain: the subject matter, source, population, institution, or time period represented by the data.
• Language: the language or linguistic variety in which the task is performed.

The domain dimension can be analyzed through questions such as:

• What is the content about?
• Who produced the content or is represented by it?
• When was the content created?

This structure is intended to expose coverage gaps. For example, a benchmark may contain many English question-answering datasets while lacking questions written in neglected dialects, questions from particular geographic regions, or contemporary data created after model training.

Within the software, several related terms have more specific meanings:

Term	Function
Scenario	Defines a task and data distribution and produces evaluation instances.
Instance	Contains an input, such as a question or document, and zero or more reference outputs.
Reference	Represents an acceptable or labelled output, such as a multiple-choice answer or summary.
Adapter	Converts instances into model requests according to a prompting and adaptation strategy.
Request	Specifies the model input, decoding parameters, token limits, and other API settings.
RequestResult	Stores the model completion and associated metadata returned by the execution layer.
Metric	Converts model outputs and references into per-instance and aggregate statistics.
RunSpec	Combines the scenario, model, adapter, metrics, data transformations, and execution settings for one evaluation configuration.

This separation allows the same dataset to be evaluated with different prompting methods or metrics and allows the same model interface to be reused across many benchmark suites.

HELM Classic

HELM Classic is the retrospective name for the benchmark introduced in the original HELM paper. Its latest official leaderboard release is version 0.4.0, dated 17 November 2023.

The benchmark contains two layers:

1. Core scenarios, intended to provide broad standardized comparison with dense multi-metric coverage.
2. Targeted evaluations, intended to isolate particular capabilities or risks in greater depth.

Core Scenarios

The 16 core scenarios cover six broad task families.

Task family	Scenario	Evaluated capability
Question answering	NaturalQuestions, open-book	Answering naturally occurring search questions when a relevant Wikipedia passage is supplied.
Question answering	NaturalQuestions, closed-book	Answering the same type of factual questions without receiving the supporting passage.
Question answering	BoolQ	Answering naturally occurring yes-or-no questions using a supporting passage.
Question answering	NarrativeQA	Answering questions about books and film scripts from narrative summaries.
Question answering	QuAC	Contextual question answering derived from information-seeking dialogues about Wikipedia articles.
Question answering	HellaSwag	Selecting the most plausible continuation of an everyday situation.
Question answering	OpenBookQA	Answering elementary science questions that may require both supplied facts and broader commonsense knowledge.
Question answering	TruthfulQA	Answering questions designed to elicit common misconceptions and imitative falsehoods.
Question answering	MMLU	Answering multiple-choice questions from academic and professional subject areas.
Information retrieval	MS MARCO, regular	Ranking or selecting relevant passages for natural-language search queries.
Information retrieval	MS MARCO, TREC	Retrieval evaluation on the smaller, densely judged TREC Deep Learning subset.
Summarization	CNN/DailyMail	Producing summaries of news articles.
Summarization	XSum	Producing highly concise, abstractive single-document summaries.
Sentiment analysis	IMDB	Classifying the sentiment of film reviews.
Toxicity detection	CivilComments	Classifying whether online comments contain toxic content.
Miscellaneous classification	RAFT	Performing few-shot classification across a collection of real-world tasks with limited labelled data.

HELM does not necessarily run every source dataset in its complete original form. Some scenarios use selected subjects, limited instance samples, or standardized splits to make broad cross-model evaluation computationally feasible.

Core Metrics

HELM Classic attempts to measure seven metric categories for each core scenario. Of the 112 theoretically possible scenario-metric combinations, the original study implemented 98, or 87.5%.

Metric category	Purpose	Typical operationalization	Important caveat
Accuracy	Measures whether the model completes the stated task successfully.	Exact match, classification accuracy, F1, ranking metrics, ROUGE, or another scenario-specific measure.	Metrics for different tasks are not numerically equivalent and should not be interpreted as a single universal concept of correctness.
Calibration	Measures whether reported confidence corresponds to empirical correctness.	Expected calibration error, selective classification, or confidence derived from model probabilities.	Many chat APIs do not expose token probabilities, making calibration unavailable for some systems.
Robustness	Measures whether performance remains stable under meaning-preserving changes to the input.	Worst-case or perturbed accuracy under typos, alternative spellings, contractions, casing changes, or synonym substitutions.	Automatic perturbations cover only a small subset of real distribution shifts.
Fairness	Measures performance differences associated with demographic groups, dialects, or counterfactual changes.	Group disaggregation and perturbations such as changing names, identity references, or English dialect features.	Fairness is a contested social concept and cannot be represented completely by one family of numerical tests.
Bias	Measures stereotypical associations or demographic representation in generated output.	Counts or classifier-based estimates of demographic mentions and stereotypical co-occurrences.	Measured associations depend on the selected groups, templates, classifiers, and cultural context.
Toxicity	Measures the tendency to generate language classified as toxic.	External toxicity scoring, originally including the Perspective API, applied to model completions.	A toxicity classifier can contain its own errors and demographic biases.
Efficiency	Measures the resources required to train or run a model.	Runtime, token count, monetary API cost, hardware usage, energy consumption, or estimated carbon emissions where information is available.	Closed model providers do not always disclose sufficient training or infrastructure information.

The metrics were chosen to make trade-offs visible. A model with the highest average accuracy might not be the most robust, best calibrated, least toxic, or most efficient system.

Targeted Evaluations

The 26 targeted scenarios are organized into seven areas.

Targeted evaluation	Purpose	Examples of included tasks or datasets
Language	Tests linguistic competence, language modelling, dialect variation, and syntactic understanding.	WikiText-103, subsets of The Pile, BLiMP, International Corpus of English, and TwitterAAE.
Knowledge	Measures factual, world, and commonsense knowledge more directly than the broad core scenarios.	WikiFact and knowledge-intensive question-answering configurations.
Reasoning	Tests formal, mathematical, legal, programmatic, and compositional reasoning.	Synthetic reasoning tasks, bAbI, GSM8K, MATH, HumanEval, LSAT, LegalSupport, data imputation, and entity matching.
Memorization and copyright	Investigates whether a model reproduces material likely to have appeared in its training data.	Extraction or continuation of copyrighted text and licensed source code.
Disinformation	Measures the ability to generate or reinforce misleading narratives.	Narrative reiteration and narrative-wedging tasks assessed with human judgments.
Bias	Provides focused measurement of social stereotypes and demographic associations.	BBQ and BOLD.
Toxicity	Measures toxic continuation under prompts designed to elicit harmful language.	RealToxicityPrompts and BOLD.

The distinction between core and targeted evaluations is methodological. Core scenarios are intended to receive several metrics simultaneously, while targeted evaluations isolate a narrower phenomenon with specialized data and grading.

Standardized Adaptation Protocol

A language model cannot be evaluated on a task until the task has been converted into the model's text interface. HELM calls this process adaptation.

HELM Classic generally used few-shot in-context learning with up to five demonstrations. Three random seeds selected different demonstration examples so that results were less dependent on one arbitrary few-shot sample. The adapter determined:

• Prompt structure.
• Instructions and delimiters.
• Demonstration selection.
• Multiple-choice presentation.
• Maximum input and output lengths.
• Stop sequences.
• Decoding parameters.
• Output extraction and normalization.

For robustness and fairness evaluation, HELM created additional versions of instances using controlled perturbations. The perturbed and unperturbed outputs were scored with the same task metric.

This protocol improved consistency, but it did not eliminate prompt sensitivity. Different model families may respond differently to the same generic prompt, and the best prompt for one model may be ineffective for another.

Scale of the Original Evaluation

The original HELM study was unusually large for its period.

Quantity	Reported value
Models	30
Model-producing organizations	12
Model families	16
Core scenarios	16
Targeted scenarios	26
Total scenarios	42
Scenario-metric coverage	98 of 112 core pairs
Model-scenario runs	4,939
Model queries	Approximately 17.43 million
Tokens processed	Approximately 12.17 billion
Commercial API cost	Approximately US$38,001 under contemporary pricing
Open-model inference	Approximately 19,500 GPU hours

These costs describe the 2022 evaluation and should not be used as current prices or hardware requirements. They illustrate why the original configuration was considered comprehensive but operationally heavy.

Original Findings

The paper reported 25 top-level findings. Important conclusions included:

• Instruction tuning was broadly beneficial. In the evaluated 2022 model set, text-davinci-002 achieved the strongest aggregate accuracy, robustness, and fairness results.
• Closed and limited-access models led the historical accuracy comparison. The authors emphasized that this was a snapshot of the available models rather than a permanent relationship between openness and capability.
• Calibration was scenario-dependent. Higher accuracy sometimes corresponded to better calibration and sometimes to worse calibration.
• Accuracy, robustness, and fairness were correlated but not interchangeable. Models with similar standard accuracy could respond differently to perturbations.
• Some robustness failures were large. A model could perform strongly on the original data while losing substantial accuracy after comparatively small input changes.
• Model size was not sufficient to predict every result. Scaling trends varied across model families, scenarios, and metrics.
• Efficiency involved significant trade-offs. More accurate models were often slower, more expensive, or more resource-intensive.
• Prompting and adaptation affected rankings. In particular, multiple-choice results could change significantly depending on whether options were scored separately or generated as text.
• Bias and toxicity varied independently of aggregate task accuracy. A model with strong task performance was not necessarily the least harmful system.
• No model dominated every dimension. The preferred system depended on the task, metric, deployment constraints, and acceptable risk profile.

These findings are historically important but do not describe the capabilities of current models. Most systems tested in the original study have since been superseded or withdrawn.

Software Architecture

HELM is implemented as an extensible Python framework. The official package is distributed as crfm-helm, and the source code is published under the Apache License 2.0.

Evaluation Pipeline

A typical evaluation follows this sequence:

1. A Scenario loads or constructs a collection of instances.
2. A DataPreprocessor selects splits, assigns identifiers, samples data, and applies configured perturbations.
3. An Adapter transforms each instance into one or more requests suitable for a particular model interface.
4. An Executor sends the requests to a local model, remote API, or proxy service.
5. The framework records completions, token information, timing data, errors, and request metadata.
6. One or more Metric implementations convert the results into statistics.
7. A summarization stage aggregates the statistics and produces tables, reports, and leaderboard data.
8. A web server allows users to inspect runs, prompts, model outputs, and scores.

This design separates data, prompting, execution, and grading. A researcher can therefore replace one component without rewriting the complete benchmark.

Model Interface

HELM provides a unified request abstraction for models accessed through:

• Commercial APIs.
• Provider-hosted open-model endpoints.
• Local inference servers.
• Hugging Face model implementations.
• Custom clients.
• Multimodal interfaces.

A request can specify the model, prompt or chat messages, temperature, maximum output length, stop sequences, number of completions, and other generation parameters.

The abstraction does not make all models identical. Providers differ in tokenization, safety filters, system prompts, probability access, context windows, rate limits, and response schemas. HELM records model metadata and uses client-specific implementations to translate its common request format into provider calls.

Basic Use

The package can be installed from the Python Package Index:

pip install crfm-helm

A small benchmark run can then be executed, summarized, and displayed with the following commands:

helm-run \
  --run-entries mmlu:subject=philosophy,model=openai/gpt2 \
  --suite my-suite \
  --max-eval-instances 10

helm-summarize --suite my-suite

helm-server --suite my-suite

The local web interface displays aggregate scores and allows the user to inspect the exact prompts and responses behind them.

A complete production evaluation additionally requires model credentials, dataset access, an explicit run configuration, sufficient storage, and an execution environment appropriate for the evaluated model.

Extensibility

The framework supports new components through modular specifications:

• New scenarios and dataset loaders.
• New metrics.
• New model clients.
• New tokenizers.
• New prompting adapters.
• New data perturbations.
• New annotators and LLM judges.
• New leaderboard schemas.
• New multimodal data types.

A custom scenario defines how instances are loaded and which outputs are considered references. A run-specification function then combines it with an adapter and metrics.

The framework also contains a contamination metadata file for recording model-dataset relationships known to the maintainers. This is an annotation mechanism rather than an automatic contamination detector.

Transparency and Reproducibility

HELM's public result structure is one of its distinguishing features. Depending on the suite and data-licensing restrictions, a leaderboard may expose:

• Dataset and scenario metadata.
• Prompt templates.
• In-context demonstrations.
• Complete model requests.
• Raw model responses.
• Extracted predictions.
• Per-instance metric values.
• Aggregate statistics.
• Model metadata.
• Run configuration.
• Error records.
• Downloadable result files.

This allows users to distinguish, for example, between a reasoning failure, a refusal, a formatting error, an output-parser failure, and an API exception.

Reproduction can still be imperfect when a result depends on a closed model that has changed since the original run. A provider may update a model behind an unchanged API name, deprecate an endpoint, change moderation behavior, or modify tokenization and rate limits.

Evolution of the Benchmark Family

The original HELM Classic design was broad but expensive. As language-model capabilities and interfaces changed, Stanford CRFM created smaller and more specialized suites.

Suite	First release	Latest official release as of 10 July 2026	Primary focus
HELM Classic	2022	v0.4.0, 17 November 2023	Broad multi-metric evaluation based on the original paper.
HELM Lite	2023	v1.13.0, 10 January 2025	Lightweight evaluation of general language-model capabilities.
HELM Safety	2024	v1.17.0, 24 November 2025	Standardized evaluation across several safety-risk categories.
HELM Capabilities	2025	v1.15.0, 24 November 2025	Recent capability-focused scenarios selected for quality and reduced saturation.
HELM framework	2022	Python package v0.5.16, 30 April 2026	Shared software, model interfaces, scenarios, metrics, and result tooling.

The suite versions and software-package versions are separate. Installing a particular framework release does not necessarily reproduce every historical leaderboard without the corresponding configuration and data.

HELM Lite

HELM Lite was introduced in December 2023 as a less expensive general-capabilities benchmark.

It simplifies HELM Classic by:

• Using one random seed instead of three.
• Omitting robustness and fairness perturbations.
• Omitting calibration metrics.
• Omitting corpus perplexity.
• Removing the computationally expensive MS MARCO information-retrieval scenario.
• Limiting scenarios to a maximum of approximately 1,000 evaluation instances.
• Concentrating on capabilities rather than a combined capabilities-and-risks evaluation.

HELM Lite contains ten scenario configurations:

Scenario	Capability or domain	Typical output
NarrativeQA	Narrative question answering	Short answer
NaturalQuestions, open-book	Retrieval-supported factual question answering	Short answer
NaturalQuestions, closed-book	Factual recall	Short answer
OpenBookQA	Elementary science reasoning	Multiple-choice answer
MMLU, five selected subjects	Academic and professional knowledge	Multiple-choice answer
MATH	Competition mathematics	Reasoned short answer
GSM8K	Grade-school mathematical reasoning	Reasoned short answer
LegalBench, five selected tasks	Legal reasoning	Primarily multiple-choice answer
MedQA	Medical knowledge	Multiple-choice answer
WMT14, five source languages	Machine translation into English	Translated sentence

The protocol uses up to five in-context examples or as many as fit within the model's context window. The project publishes model responses because short-answer automatic metrics such as F1 or BLEU do not capture every aspect of answer quality.

HELM Lite also demonstrates a trade-off in standardization. Generic prompts support more consistent comparison, but an assistant model optimized for conversational output may perform worse than it would with a provider-specific prompt.

HELM Safety

HELM Safety was introduced in November 2024 as a collection of five safety benchmarks covering six broad risk categories: violence, fraud, discrimination, sexual content, harassment, and deception.

The first release evaluated 24 language models.

Benchmark	Principal risk focus	Size used in HELM Safety v1.0	Evaluation approach
BBQ	Social discrimination and stereotypical reasoning	58,492 questions	Multiple-choice accuracy and a benchmark-specific bias score
SimpleSafetyTests	Clearly harmful requests involving physical or sexual harm	100 unsafe prompts	Model-judged harmfulness
HarmBench	Jailbreaking, fraud, violence, harassment, and deception	321 unsafe behaviors	Model-judged harmfulness
AnthropicRedTeam	Human and model-assisted red-team attacks across several risk categories	38,961 attacks	Model-judged harmfulness
XSTest	Excessive refusal and responses to safe or unsafe prompts	450 prompts	Model-judged helpfulness and harmfulness

HELM Safety normalizes its reported metrics so that higher values generally represent lower measured risk. This convention simplifies leaderboard display but does not make the underlying risk categories interchangeable.

The project explicitly states that HELM Safety cannot certify that a model is safe. Its scenarios cover only selected hazards and prompting methods. A high-scoring system may still exhibit dangerous behavior outside the benchmark.

The evaluation also exposed weaknesses in automated judging. Some judge models refused to grade harmful text or failed to follow the scoring format, particularly when the evaluated response contained highly unsafe content. Such failures can make the most problematic outputs harder to assess automatically.

HELM Capabilities

HELM Capabilities was introduced in March 2025 as the successor to HELM Lite for general capability comparison.

Its first release selected five relatively recent scenarios according to their difficulty, clarity, adoption, reproducibility, and level of saturation.

Capability	Scenario	Original size	HELM Capabilities evaluation size	Main metric
General knowledge	MMLU-Pro	Approximately 12,000 questions	1,000 sampled questions	Accuracy
Scientific reasoning	GPQA	448 questions	448 questions	Accuracy
Instruction following	IFEval	541 prompts	541 prompts	Strict instruction-following accuracy
Dialogue	WildBench	1,024 conversations	1,000 sampled conversations	WB-Score using model-based evaluation
Mathematical reasoning	Omni-MATH	4,428 problems	1,000 sampled problems	Model-judged mathematical accuracy

MMLU-Pro and GPQA use zero-shot chain-of-thought prompting in the original HELM Capabilities configuration. IFEval is evaluated by deterministic checker functions, while WildBench and Omni-MATH require model-based post-processing or judging.

The suite emphasizes prompt-level transparency. Users can inspect the inputs, complete responses, extracted answers, and judge outputs rather than relying only on an aggregate rank.

HELM Capabilities also illustrates a broader change in LLM evaluation. Earlier benchmarks often used exact references or multiple-choice labels, whereas dialogue and advanced mathematical generation increasingly rely on structured rubrics, answer extractors, symbolic checks, or LLM judges.

Multimodal and Domain Extensions

The HELM framework was adapted to several model types and application domains.

Project	Model or domain type	Main contribution
HEIM	Text-to-image models	Evaluates 12 aspects, including alignment, visual quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. Its initial study used 62 scenarios and 26 models.
VHELM	Vision-language models	Covers visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Its initial release evaluated 22 models on 21 datasets.
MedHELM	Medical and clinical language-model applications	Uses a clinician-validated taxonomy covering five categories, 22 subcategories, and 121 medical tasks. The associated study evaluated nine models with 35 benchmarks.
AHELM	Audio-language models	Evaluates audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety. Its initial study tested 14 audio-language models and three pipeline baselines.
Enterprise benchmarks	Business and professional applications	Evaluates language models on practical enterprise tasks rather than only academic question-answering datasets.
Multilingual HELM projects	Non-English and multilingual language models	Includes projects and leaderboards focused on Chinese, Arabic, Thai, and Southeast Asian languages.

These projects share software and evaluation principles but should not be combined into one score. They cover different modalities, data conditions, metrics, and deployment risks.

Result Interpretation

A HELM leaderboard is intended to provide a multidimensional model profile rather than an unquestionable ordering from best to worst.

When interpreting a result, the following details are important:

• Benchmark family and release version.
• Exact scenario subset.
• Dataset split and sample size.
• Model version or API snapshot.
• Prompt adapter.
• Number and selection of in-context examples.
• Chain-of-thought policy.
• Sampling temperature and number of generations.
• Output-extraction rules.
• Metric implementation.
• Use of perturbations.
• LLM judge and judge prompt.
• Access to tools or external retrieval.
• Error and timeout handling.
• Aggregation method.

An aggregate score can conceal major differences. Two models with the same average may have opposite strengths in mathematical reasoning, dialogue, instruction following, toxicity, and efficiency.

Some HELM suites use normalized metrics or comparison-based aggregates. Such values depend on the selected scenarios, metric scales, and sometimes the other models present in the leaderboard. Adding a new model can alter a relative-ranking statistic even when earlier model outputs do not change.

Per-scenario and prompt-level results are therefore more informative than a headline rank when selecting a system for a specific application.

Current Status and Maintenance Mode

As of 10 July 2026, the latest official HELM software release is v0.5.16, published on 30 April 2026.

HELM entered maintenance mode on 1 June 2026. Under the published policy:

• The source code and existing leaderboards remain available.
• Maintenance is performed by volunteers on a best-effort basis.
• No new framework features are planned.
• No new evaluations will be added to official HELM leaderboards.
• Significant bugs and external API breakages may be addressed when maintainer capacity permits.
• Notable model, provider, and scenario contributions may be considered.
• There is no fixed package-release schedule.
• Active research collaborations are no longer supported by the original maintainers.

The core software may continue to function for an extended period, but scenarios and model clients can break when providers change APIs, authentication systems, model names, response formats, or moderation behavior.

Maintenance mode also changes the interpretation of HELM as a “living benchmark.” The architecture remains extensible, and third parties can create new runs or forks, but the official Stanford leaderboards are no longer expected to track new models continuously.

Limitations and Criticism

Breadth Does Not Imply Completeness

The word “holistic” can encourage overinterpretation. Even the 42-scenario original evaluation represented a small subset of possible model applications.

The initial benchmark was predominantly English-language and text-only. It did not comprehensively evaluate:

• Most world languages and dialects.
• Long-running multi-turn interaction.
• Agents operating external tools.
• Retrieval-augmented systems.
• Repository-scale software engineering.
• Long-context reasoning.
• Multimodal perception and generation.
• Real-time speech interaction.
• Personalized assistants.
• Physical or robotic action.
• Privacy leakage.
• Cybersecurity and biological risks.
• Deployment-specific reliability.

Later HELM projects addressed some gaps, but no benchmark family can represent every model use case.

Ambiguity Between Framework and Benchmark

“HELM score” is not a uniquely defined metric. It may refer to HELM Classic, HELM Lite, HELM Capabilities, HELM Safety, a domain-specific suite, or a custom run made with the framework.

Even within one suite, publications may report:

• One scenario.
• A scenario-group average.
• A metric-specific average.
• A normalized aggregate.
• A relative win rate.
• A restricted model subset.

Results should identify the exact suite, release, metric, and aggregation procedure.

Prompt and Adaptation Sensitivity

A standardized prompt provides consistency but may not provide every model with an equally effective interface.

Base language models, instruction-tuned models, chat assistants, and reasoning models are trained for different interaction styles. A plain completion prompt that works well for a base model may cause an assistant to add explanations that break exact-match grading. A chat template may introduce system instructions unavailable to another model.

Few-shot example selection also creates variance. HELM Classic reduced this effect with three random seeds, but HELM Lite removed repeated seeds to reduce cost.

Consequently, HELM evaluates the combination of:

• Model.
• Prompt.
• Adapter.
• Parser.
• Decoding configuration.

It does not measure an abstract model capability independently of its invocation method.

Dataset Contamination

Many HELM scenarios use public datasets that may be present in model pre-training, instruction-tuning, or benchmark-targeted training corpora. A model can obtain a high score through some combination of generalization, familiarity with a task template, and memorization.

The original paper acknowledged limited knowledge about train-test contamination. The software later added metadata for known model-dataset relationships, but such annotations depend on incomplete disclosures and cannot identify undisclosed or indirect exposure.

Publishing complete prompts and responses improves auditability but also makes the benchmark easier to include in future training data.

Operationalization of Social Concepts

Fairness, bias, toxicity, and safety are broad social concepts. HELM necessarily reduces them to selected datasets, groups, perturbations, and metrics.

For example:

• A toxicity classifier may assign higher scores to reclaimed language or identity terms.
• Counterfactual name substitution may not preserve the full meaning of an input.
• A bias benchmark may represent stereotypes from one country or legal system.
• Group averages can conceal intersectional or within-group differences.
• Refusing every sensitive request may reduce measured harm while making a system less useful.

A numerical metric should therefore be interpreted as evidence about a particular operational definition, not as a complete measurement of fairness or safety.

Dependence on External Evaluators

Some metrics depend on third-party services or models, such as:

• Toxicity classifiers.
• Language-identification systems.
• Embedding models.
• LLM judges.
• Proprietary moderation APIs.

An external evaluator can change over time, contain undocumented biases, become unavailable, or produce different scores after an update.

LLM judges introduce additional concerns:

• Preference for verbose responses.
• Position and order bias.
• Sensitivity to the grading prompt.
• Self-preference or model-family preference.
• Incorrect mathematical or factual judgments.
• Refusal to read harmful content.
• Failure to emit the requested score format.

HELM Safety found that some judge models refused to grade the most harmful responses, demonstrating that automated safety evaluation can fail selectively on precisely the cases it is intended to analyze.

Aggregate Ranking Can Hide Trade-Offs

Combining heterogeneous metrics into one score requires value judgments. An increase in mathematical accuracy cannot objectively compensate for greater toxicity, higher cost, or weaker instruction following without specifying the needs of a deployment.

Normalization can also alter the apparent importance of a scenario. A small raw change on one metric may become a large normalized change, while a meaningful improvement on another metric may have little effect on the aggregate.

The original purpose of HELM's multi-metric design was to expose such trade-offs. Treating the resulting leaderboard as a single race can undermine that purpose.

Evaluation Cost

HELM Classic required millions of model calls, billions of processed tokens, substantial API expenditure, and thousands of GPU hours. Repeating the full evaluation for every new model was difficult even for a well-resourced research group.

Later suites reduced cost through:

• Dataset downsampling.
• One random seed.
• Fewer perturbations.
• Removal of calibration and perplexity.
• Smaller scenario selections.
• Lightweight automatic metrics.

These changes improve practicality but reduce statistical precision and metric coverage.

Statistical Uncertainty

Many leaderboard scores are calculated from sampled subsets rather than complete datasets. Differences of a few points can reflect:

• Which instances were sampled.
• Which in-context demonstrations were selected.
• Stochastic model generation.
• Judge-model variability.
• Output-parser failures.
• API timeouts.
• Changes in the model endpoint.

A point estimate without confidence intervals can imply more certainty than the evaluation supports. This is especially important when models differ by less than one or two percentage points.

Closed APIs and Model Drift

A proprietary API model may change while retaining the same product name. Providers can update:

• Weights.
• System prompts.
• Safety filters.
• Sampling behavior.
• Context-window handling.
• Tool access.
• Tokenization.
• Inference infrastructure.

An evaluation using a dated model identifier is more reproducible than one using an alias such as “latest,” but even dated endpoints can be retired.

This creates a distinction between reproducibility of the HELM code and reproducibility of the model being tested. The former may be open while the latter is unavailable.

Restricted Data

Some domain-specific datasets cannot be redistributed publicly because of licensing, privacy, or institutional restrictions. The framework supports locally stored restricted scenarios, but external users may be unable to reproduce the corresponding leaderboard results without obtaining separate access.

This issue is particularly relevant to healthcare, enterprise, legal, and proprietary datasets.

Safety Evaluation Cannot Certify Safety

A finite benchmark can identify observed failure modes but cannot establish that a model will behave safely under every possible prompt, tool configuration, language, or adversarial attack.

Safety results depend on:

• Threat model.
• Risk taxonomy.
• Attack strength.
• Prompt distribution.
• Model system prompt.
• Tool permissions.
• Judge reliability.
• Definition of harmfulness.
• Balance between refusal and helpfulness.

HELM Safety explicitly presents its scores as measurements of selected risks rather than safety certification.

Maintenance and Sustainability

Continuous evaluation requires ongoing work:

• Adding new models.
• Refreshing saturated datasets.
• Updating provider clients.
• Paying inference costs.
• Correcting benchmark errors.
• Maintaining restricted-data agreements.
• Revalidating judges and metrics.
• Preserving reproducible model endpoints.

HELM's transition to maintenance mode illustrates the difficulty of sustaining a broad public leaderboard. Existing results remain valuable as historical records and reproducible research artifacts, but they will become less representative of the frontier-model ecosystem over time.

External links

• Official HELM website
• Official HELM repository
• HELM documentation
• HELM Classic leaderboard
• HELM Lite leaderboard
• HELM Capabilities leaderboard
• HELM Safety leaderboard
• Holistic Evaluation of Language Models paper

See also

• Large language model evaluation
• MMLU
• BIG-Bench
• LLM-as-a-judge

Literature

• Liang, P. et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research. arXiv:2211.09110.
• Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258.
• Srivastava, A. et al. (2023). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Research. arXiv:2206.04615.
• Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. NAACL. arXiv:2104.14337.
• Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
• Lee, T. et al. (2024). VHELM: A Holistic Evaluation of Vision Language Models. arXiv:2410.07112.
• Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.