IFEval Benchmark

IFEval (Instruction-Following Eval) is a benchmark for measuring how precisely large language models (LLMs) follow explicit natural-language instructions. It was introduced by Jeffrey Zhou and colleagues in the 2023 paper Instruction-Following Evaluation for Large Language Models.

IFEval focuses on verifiable instructions: constraints whose satisfaction can be checked by a simple and interpretable computer program. Examples include requiring a response to contain specified keywords, use a particular language, remain below a word limit, omit commas, contain a fixed number of bullet points, or consist entirely of valid JSON.

The public benchmark contains 541 prompts and 25 instruction types. Each prompt contains one to three constraints combined with an open-ended user task. The prompt text is in English, although some tasks require the response to be written in another language. The released data include the applicable instruction identifiers and checker parameters, while the official evaluator applies deterministic rule-based tests to the generated response.

Unlike knowledge and reasoning benchmarks such as MMLU, BIG-Bench Hard, or MATH-500, IFEval does not primarily determine whether the substantive content of an answer is factually correct. Its principal purpose is to determine whether the response obeys explicitly stated, automatically testable output requirements.

Background and Purpose

Instruction following is a central capability of instruction-tuned and conversational language models. A model may possess the required knowledge yet still produce an unusable answer because it ignores the requested format, exceeds a length limit, omits a required element, adds prohibited text, or responds in the wrong language.

Before IFEval, instruction-following quality was commonly assessed through human review, pairwise preference judgments, or evaluation by another language model. Human evaluation can be costly and difficult to reproduce consistently, while an LLM judge can inherit its own biases and reasoning errors. Conventional reference-based metrics are also poorly suited to open-ended prompts for which many different responses may be acceptable.

IFEval narrows the evaluation problem to a reproducible question:

Does the model response satisfy every verifiable constraint stated in the prompt?

This design has several practical advantages:

• Grading is automatic and inexpensive after responses have been generated.
• The same response receives the same result under a fixed checker implementation.
• No complete reference answer is required.
• Failures can be analyzed by instruction type.
• Open-ended generation can be evaluated without reducing every task to multiple-choice selection.
• Checker outcomes can be used as automatically computed rewards in training experiments.

The benchmark does not attempt to replace human evaluation of relevance, correctness, safety, usefulness, or style. It measures a narrower capability usually described as precise instruction following.

Dataset Construction

The authors constructed the prompts through four stages:

1. A base task was combined with one to three randomly selected verifiable instructions.
2. A few-shot language-model procedure was used to identify and remove illogical or incompatible combinations.
3. Another few-shot procedure rephrased prompts to increase linguistic diversity.
4. Every prompt was manually reviewed and edited before inclusion.

Instruction parameters were varied across examples. A length constraint could require at least or fewer than a specified number of words. A keyword constraint could require a phrase to appear, prohibit it, or specify its frequency. Formatting instructions could determine the number of paragraphs, sections, bullet points, placeholders, or highlighted passages.

Property	Value
Number of prompts	541
Number of instruction types	25
Constraints per prompt	One to three
Prompt language	English
Interaction format	Single-turn text generation
Response format	Open-ended text subject to verifiable constraints
Grading	Deterministic instruction-specific checker functions
Conventional gold response	Not required

IFEval is an evaluation set rather than a conventional supervised-learning corpus. The Hugging Face distribution exposes the records in a split named train, but the 541 items are intended for benchmark evaluation rather than model training.

Instruction Taxonomy

The original paper groups the 25 instruction types into nine broad categories.

Group	Instruction type	Official identifier	Requirement
Keywords	Include keywords	keywords:existence	One or more specified words or phrases must appear.
Keywords	Keyword frequency	keywords:frequency	A specified keyword must occur according to a required frequency.
Keywords	Forbidden words	keywords:forbidden_words	Specified words or phrases must not appear.
Keywords	Letter frequency	keywords:letter_frequency	A specified letter must occur according to a numerical condition.
Language	Response language	language:response_language	The complete response must be written in a specified language.
Length constraints	Number of sentences	length_constraints:number_sentences	The sentence count must satisfy a minimum or maximum threshold.
Length constraints	Number of paragraphs	length_constraints:number_paragraphs	The response must contain the required number of paragraphs.
Length constraints	Number of words	length_constraints:number_words	The word count must satisfy a minimum or maximum threshold.
Length constraints	First word of a specified paragraph	length_constraints:nth_paragraph_first_word	A selected paragraph must begin with a specified word.
Detectable content	Number of placeholders	detectable_content:number_placeholders	The response must contain a required number of bracketed placeholders.
Detectable content	Postscript	detectable_content:postscript	A postscript beginning with a specified marker must be included.
Detectable format	Number of bullet points	detectable_format:number_bullet_lists	The answer must contain the requested number of Markdown bullet points.
Detectable format	Constrained response	detectable_format:constrained_response	The response must be selected from a predefined set of allowed outputs.
Detectable format	Highlighted sections	detectable_format:number_highlighted_sections	At least a specified number of passages must use the requested emphasis markup.
Detectable format	Multiple sections	detectable_format:multiple_sections	The response must contain a specified number of sections marked by a required separator.
Detectable format	JSON format	detectable_format:json_format	The complete response must parse as JSON.
Detectable format	Title	detectable_format:title	A title must be enclosed in the requested delimiters, such as <<title>>.
Combination	Two responses	combination:two_responses	Two separate answers must be divided by a specified separator.
Combination	Repeat prompt	combination:repeat_prompt	The model must reproduce specified prompt text before answering.
Change case	Capital-word frequency	change_case:capital_word_frequency	Fully capitalized words must occur according to a numerical condition.
Change case	All uppercase	change_case:english_capital	English alphabetic characters in the response must be uppercase.
Change case	All lowercase	change_case:english_lowercase	English alphabetic characters in the response must be lowercase.
Start or end	End checker	startend:end_checker	The response must end with an exact phrase and contain nothing after it.
Start or end	Quotation	startend:quotation	The entire response must be enclosed in quotation marks.
Punctuation	No commas	punctuation:no_comma	No comma character may appear in the response.

The instruction types differ in implementation complexity. Some use direct string matching or regular expressions, while others require word and sentence segmentation, language identification, JSON parsing, or task-specific structural logic. The official registry maps each identifier to a checker class in the released Python code.

Data Format and Release

The official release stores its prompts in JSONL format. Every record contains four principal fields:

Field	Description
key	A unique integer identifier for the prompt.
prompt	The complete task and its natural-language constraints.
instruction_id_list	An ordered list of checker identifiers applicable to the prompt.
kwargs	An ordered list of parameter dictionaries aligned with the instruction identifiers.

A simplified record, with unused parameters omitted, can be represented as follows:

{
  "key": 1000,
  "prompt": "Write an explanation. Use at least 300 words, highlight three sections, and do not use commas.",
  "instruction_id_list": [
    "punctuation:no_comma",
    "detectable_format:number_highlighted_sections",
    "length_constraints:number_words"
  ],
  "kwargs": [
    {},
    {"num_highlights": 3},
    {"relation": "at least", "num_words": 300}
  ]
}

The two arrays are positionally aligned: the first parameter dictionary configures the first checker, the second configures the second checker, and so forth.

There is no single canonical natural-language response. Any generated text can receive full credit if it satisfies all configured constraints. The official repository contains the prompt file, checker classes, the instruction registry, strict and loose evaluation functions, a command-line runner, and instructions for supplying model responses in JSONL form.

Evaluation Methodology

Response Generation

Each benchmark prompt is sent to the evaluated model as a single user request. The generated response is stored together with the exact prompt. The official runner expects records with the following basic structure:

{
  "prompt": "The exact IFEval prompt",
  "response": "The model-generated response"
}

The evaluator uses the prompt as a key, loads the corresponding instruction identifiers and parameters, instantiates the appropriate checker classes, and evaluates each constraint separately.

Strict Evaluation

Under strict evaluation, every checker is applied directly to the complete raw response. A strict check may count words or sentences, search for required or forbidden strings, parse JSON, test capitalization, inspect the first or last characters, or verify the requested structural separators.

Each instruction receives a binary result: passed or failed. There is no partial credit within an individual instruction.

Strict evaluation intentionally penalizes additional material that violates the requested format. For example, a response containing a valid JSON object preceded by “Here is the result” fails the strict JSON-format checker because the complete response is not valid JSON.

Loose Evaluation

The authors introduced loose evaluation to reduce false negatives caused by common chat-model formatting habits. The official implementation constructs eight versions of each response:

• The original response.
• A version with asterisks removed.
• A version without the first line.
• A version without the last line.
• A version without both the first and last lines.
• Each of the three line-removal variants with asterisks also removed.

An instruction passes under the loose criterion if at least one of the eight variants passes its checker. This can accommodate a conventional introduction, closing sentence, or Markdown emphasis markers.

Loose evaluation may also create false positives because deleting text can remove a genuine violation. The original paper therefore treats loose accuracy as a complement to strict accuracy rather than as a replacement for it.

Metrics

IFEval reports results at both prompt and instruction levels.

Metric	Success condition	Parser-safe definition
Prompt-level strict accuracy	Every instruction attached to a prompt passes strict evaluation.	fully strict-compliant prompts / all prompts
Instruction-level strict accuracy	Each instruction instance is counted independently under strict evaluation.	strictly passed instruction instances / all instruction instances
Prompt-level loose accuracy	Every instruction attached to a prompt passes loose evaluation.	fully loose-compliant prompts / all prompts
Instruction-level loose accuracy	Each instruction instance is counted independently under loose evaluation.	loosely passed instruction instances / all instruction instances

Prompt-level accuracy is usually lower because a prompt receives no credit when even one of its constraints fails. Instruction-level accuracy preserves information about partial compliance but does not show whether the passed constraints occurred together in complete responses.

Reproducibility and Protocol Details

The checkers are deterministic for a fixed response and code version, but the end-to-end score also depends on response generation and preprocessing. A reproducible report should specify:

• The exact model and checkpoint or API version.
• The system prompt and chat template.
• Sampling temperature and decoding method.
• Maximum output-token length and stop sequences.
• Whether visible reasoning text is included in the graded response.
• Any extraction or post-processing applied before grading.
• The dataset revision and evaluator implementation.
• Which of the four IFEval metrics is reported.

These details can materially affect results. A system-generated preamble can invalidate JSON, quotation, exact-start, lowercase, or uppercase requirements. A short output limit can make a minimum-word constraint impossible to satisfy, while excessive generation can add text after an otherwise valid answer.

Original Results

The original study evaluated GPT-4 and PaLM 2 S through provider APIs in 2023. The authors reported the following results:

Results reported in the original IFEval paper

Model	Prompt-level strict	Instruction-level strict	Prompt-level loose	Instruction-level loose
GPT-4	76.89%	83.57%	79.30%	85.37%
PaLM 2 S	43.07%	55.76%	46.95%	59.11%

GPT-4 achieved higher scores on all four metrics. The paper cautioned that the two systems were not directly comparable because their model sizes and other characteristics differed substantially.

The gap between prompt-level and instruction-level accuracy illustrates the compounding effect of combined constraints. A model can satisfy most individual requirements while failing a larger fraction of complete prompts because one missed requirement causes the whole prompt to fail.

The moderate increase from strict to loose accuracy indicates that some failures were attributable to removable introductions, conclusions, or Markdown formatting, while many responses remained non-compliant after all eight transformations.

Adoption and Research Use

IFEval became widely used because it combines open-ended generation with inexpensive deterministic grading. The dataset has been incorporated into public evaluation infrastructure, including the Hugging Face Open LLM evaluation ecosystem and the EleutherAI Language Model Evaluation Harness.

The benchmark has been used in model cards, instruction-tuning studies, leaderboard evaluations, analyses of prompt sensitivity, and research on training with verifiable rewards. Because every checker produces an automatic pass or fail signal, the same general design can support supervised filtering, rejection sampling, and reinforcement learning with verifiable rewards.

Published results are not always presented using the same headline metric. Some reports use prompt-level strict accuracy, while others use instruction-level or loose accuracy. A score labelled only “IFEval” is therefore ambiguous unless the metric and evaluation harness are identified.

Extensions and Related Benchmarks

IFEval influenced several later evaluations that extend its code-verifiable approach or target limitations of the original dataset.

Benchmark	Year	Main extension	Description
Multi-IF	2024	Multi-turn and multilingual evaluation	Expands IFEval into 4,501 three-turn conversations across eight languages, using English plus seven translated languages.
M-IFEval	2025	Language-specific multilingual constraints	Extends evaluation to French, Japanese, and Spanish with both general constraints and instructions designed around language-specific orthographic or grammatical properties.
IFBench	2025	Generalization to unseen constraint types	Introduces 58 new out-of-domain verifiable constraints and releases 29 additional training constraints with verification functions for studying precise instruction-following generalization.
IFEval++	2026	Reliability across nuanced prompt variants	Expands each IFEval case into a ten-prompt family and measures whether a model succeeds consistently across rephrasings, distractors, and task or constraint reconfigurations.
IFEval-FC	2025	Function-calling formats	Applies verifiable format requirements to function-call parameters and contains 750 test cases with automatically checkable outputs.

IFEval++ and Nuance-Oriented Reliability

IFEval++ was introduced to test whether high single-prompt accuracy remains stable when the same underlying requirement is expressed in slightly different ways. For each of the 541 original cases, the authors retained the original prompt and generated nine related “cousin prompts” using three methods:

• Rephrasing, which changes wording while preserving the task and constraint semantics.
• Distractor addition, which adds compatible but irrelevant contextual or instructional material.
• Task or constraint reconfiguration, which changes a parameter or presents the same requirement in a different task setting.

The result contains 541 ten-prompt groups, or 5,410 prompts in total.

IFEval++ introduced the metric reliable@k. A group receives credit only when the model succeeds on all k related prompts:

reliable@k = groups passed on all k cousin prompts / all evaluated groups

Under this definition, reliable@1 corresponds to ordinary single-prompt accuracy, while reliable@10 requires consistent success across the complete ten-prompt family.

Selected results from the IFEval++ study illustrate the difference between ordinary accuracy and group-level reliability:

Model	IFEval accuracy	IFEval++ reliable@10	Relative decline
GPT-5	95.9%	78.4%	18.3%
LLaMA-3.3-70B-Instruct	92.1%	71.0%	22.9%
Qwen3-0.6B	58.0%	22.2%	61.8%

These values are results from the paper's specific model versions and evaluation configuration, not a universal or continuously updated leaderboard. They show that near-ceiling performance on the original fixed prompts does not necessarily imply consistent instruction following under nuanced reformulations.

Limitations and Criticism

Limited Construct Coverage

IFEval measures a deliberately narrow aspect of instruction following. Its checkers focus on properties that can be observed and verified in the final text.

The benchmark does not directly evaluate:

• Factual accuracy.
• Logical validity.
• Completeness of the requested task.
• Relevance to the user's underlying goal.
• Quality of explanation.
• Citation correctness.
• Safety or policy compliance.
• Honesty about uncertainty.
• Creativity or rhetorical effectiveness.

A response can satisfy every formal constraint while containing false or irrelevant content. Conversely, a useful and accurate answer can receive no credit because it contains one prohibited comma or misses an exact word-count threshold. IFEval should therefore be interpreted as a measure of constraint adherence rather than overall assistant quality.

Checker Definitions and Edge Cases

Natural-language concepts such as “word,” “sentence,” “paragraph,” “title,” or “highlighted section” do not have universally agreed computational definitions. Results can depend on tokenization, punctuation handling, Unicode characters, Markdown syntax, abbreviations, code blocks, mathematical notation, or mixed-language text.

The original paper explicitly notes that even apparently verifiable instructions have edge cases. Its loose evaluator was introduced because simple matching can reject semantically compliant outputs that contain common formatting variations.

A model may reasonably interpret a request differently from the checker and be marked wrong. It may also exploit a checker implementation and pass without satisfying the intended human interpretation.

Data Quality and Versioning

The IFEval++ authors reported that manual review and community feedback identified a small number of flawed original cases. The reported issue types included checker limitations, mismatches between prompt wording and stored evaluation parameters, typographical inconsistencies, and combinations of constraints that could not be satisfied simultaneously. Their experiments used a minimally revised copy of the data and evaluation functions.

This creates a versioning issue: scores obtained from the original Google data, a corrected downstream copy, or a modified evaluation harness may differ. Benchmark reports should identify the exact dataset revision and code commit.

Public and Static Test Data

The complete prompts, instruction identifiers, parameters, and verifier code are publicly available. This transparency supports reproduction and error analysis, but it also creates a risk that the examples or their recurring templates may enter pre-training, instruction-tuning, preference, or reinforcement-learning data.

A high score cannot by itself distinguish general instruction-following ability from direct memorization, familiarity with the 25 checker families, or benchmark-specific optimization. IFBench was created partly because models that perform strongly on familiar verifiable constraints may generalize poorly to previously unseen constraint types.

English and Single-Turn Scope

The original prompts are written in English and each record is evaluated as an independent single-turn interaction. The benchmark does not comprehensively measure whether a model can follow instructions written in other languages, preserve requirements across dialogue turns, revise only part of an earlier request, or resolve conflicts between persistent and newly introduced constraints.

Multi-IF and M-IFEval address parts of these limitations by adding multi-turn conversations and language-specific multilingual evaluation.

Metric Aggregation

Prompt-level accuracy is all-or-nothing: satisfying two of three constraints receives the same prompt score as satisfying none. This is appropriate when complete compliance is required but discards information about partial success.

Instruction-level accuracy retains partial credit but weights instruction instances rather than prompts. Frequently occurring instruction types therefore contribute more heavily to the aggregate result, while rare but difficult constraints can have little effect on the headline score.

Per-type results, prompt-level results, and instruction-level results should be reported together where possible.

Protocol Sensitivity

IFEval scores can change because of infrastructure choices rather than a change in the underlying model. Relevant factors include system prompts, chat templates, default assistant preambles, visible reasoning text, refusal behavior, output truncation, stop sequences, response extraction, Markdown normalization, language-identification libraries, and checker revisions.

For example, a platform that automatically prefixes every response with a conversational acknowledgment makes strict JSON, quotation, uppercase, lowercase, and exact-start requirements more difficult. Removing the prefix may be operationally reasonable, but it defines a different evaluation protocol.

Saturation and Reliability

Later frontier models achieved near-ceiling results on the original benchmark in the IFEval++ study. At that point, the fixed 541-prompt average provided less resolution for distinguishing leading systems.

IFEval++ demonstrated that a high original score can coexist with materially lower reliability across related prompts. This indicates that average accuracy on familiar formulations does not fully measure robustness to rephrasing, distracting context, or changed constraint values.

For frontier-model comparisons, IFEval is therefore most informative when combined with private or newly generated cases, out-of-domain constraints, multilingual prompts, multi-turn interactions, content-quality evaluation, and per-instruction diagnostics.

External links

• Instruction-Following Evaluation for Large Language Models
• Official IFEval code and data
• IFEval dataset on Hugging Face
• IFEval in the Language Model Evaluation Harness
• IFEval++ repository
• IFBench repository

See also

• Prompt engineering
• LLM-as-a-judge

Literature

• Zhou, J. et al. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911.
• He, Y. et al. (2024). Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv:2410.15553.
• Dussolle, A. et al. (2025). M-IFEval: Multilingual Instruction-Following Evaluation. Findings of NAACL 2025. [12].
• Pyatkin, V. et al. (2025). Generalizing Verifiable Instruction Following. arXiv:2507.02833.
• Dong, J. et al. (2026). Revisiting the Reliability of Language Models in Instruction-Following. ACL 2026. [13].