LLM quality metrics

Quality metrics for large language models (LLMs) are a systematic approach and a set of standardized tools for measuring various aspects of language model performance, including accuracy, safety, fairness, and reliability. As LLMs find increasingly widespread use in critical domains such as healthcare, finance, and education, there is an urgent need for their comprehensive and objective evaluation.

Metrics and benchmarks serve several key functions: they enable objective comparisons between different models, track progress in their development, identify weaknesses, and ensure transparency of results for researchers and practitioners.

Categories of Metrics

Metrics for evaluating LLMs can be divided into several main categories: automatic metrics, human evaluation, and specialized metrics for assessing safety and reliability.

Automatic Metrics

These metrics allow for fast and scalable evaluation without human involvement.

N-gram-based Metrics

Traditional metrics that measure lexical overlap between the generated and reference text.

• BLEU (Bilingual Evaluation Understudy): Originally developed for evaluating machine translation quality. It measures the precision of n-gram matches (sequences of n words) and applies a penalty for generated texts that are too short.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Focuses on recall, measuring how well n-grams from the reference text are represented in the generated one. It is particularly effective for evaluating summarization tasks.
• METEOR: Extends the capabilities of BLEU by considering synonyms, stem words, and morphological variants, which allows for better correlation with human judgments.

Semantic Metrics

These metrics use contextual embeddings to assess semantic similarity, rather than just lexical overlap.

• BERTScore: Calculates semantic similarity between tokens in the generated and reference texts using embeddings from the BERT model. This allows it to recognize semantic equivalence even with different phrasing.
• MAUVE: Measures the divergence between the distributions of machine-generated and human-written texts in the embedding space. It is particularly effective for evaluating open-ended generation, where there is no fixed reference text.

Intrinsic Language Modeling Metrics

• Perplexity: A fundamental metric that measures how well a language model predicts a sequence of text. It reflects the model's uncertainty in predicting the next token. Lower perplexity values indicate better performance.
• Accuracy and F1-score: Widely used in classification and question-answering tasks. The F1-score is the harmonic mean of precision and recall, providing a balanced assessment.

Human Evaluation

Human evaluation remains the "gold standard," as automatic metrics often fail to capture subtle aspects of quality, such as coherence, creativity, and relevance.

• Direct Assessment: Experts or crowdworkers rate the quality of the generated output on a given scale (e.g., from 1 to 5) based on criteria like fluency and coherence.
• Comparative Assessment: Evaluators are asked to compare the outputs of two or more models and choose the best one (pairwise comparison) or rank them from best to worst.

The disadvantages of human evaluation include high cost, difficulty in scaling, and subjectivity.

Evaluation using LLMs (LLM-as-a-Judge)

A new approach where one language model (usually more powerful) is used to evaluate the responses of another. For example, GPT-4 can rank the outputs of models based on given criteria. This method provides a scalable alternative to human evaluation, although it has its own challenges, such as sensitivity to prompt style and potential biases.

Specialized Metrics and Benchmarks

Specialized metrics and benchmarks are used to evaluate specific aspects of LLM performance and reliability.

Factual Reliability

Assesses the model's ability to generate truthful information and avoid hallucinations.

• TruthfulQA: A benchmark specifically designed to measure a model's tendency to generate answers based on common myths and misconceptions. The model is required to provide factually correct answers, not just popular ones.

Safety and Ethics

• Toxicity Assessment: Measures the presence of abusive or harmful content. This is done using specialized classifiers and APIs, such as the Perspective API.
• Bias and Fairness Assessment: Evaluates whether the model exhibits discriminatory behavior toward different demographic groups. Research shows that LLMs can perpetuate and amplify social stereotypes from their training data.
• SafetyBench: A comprehensive benchmark for safety evaluation, including tests for robustness against adversarial attacks and the ability to avoid generating harmful content.

Comprehensive Benchmarks

• MMLU (Massive Multitask Language Understanding): One of the most widely used benchmarks, featuring multiple-choice questions across 57 subjects, from elementary mathematics to international law. It assesses the breadth and depth of a model's knowledge.
• BIG-bench (Beyond the Imitation Game): Contains over 204 tasks designed to evaluate capabilities that go beyond those of standard language models, including tasks ranging from playing chess to guessing emojis.

Challenges and Limitations

• Correlation Problem: Traditional automatic metrics like BLEU and ROUGE often correlate poorly with human judgments, especially in creative tasks.
• Data Contamination: There is a risk that test data from a benchmark may have been included in the model's training set, leading to inflated and unreliable scores.
• Multilingual Evaluation: Most existing metrics and benchmarks are focused on English, which limits their applicability for evaluating the multilingual capabilities of LLMs.

Links

• What Are LLM Benchmarks? — an overview article from IBM
• 20 LLM evaluation benchmarks and how they work — a guide to benchmarks from Evidently AI

Literature

• Papineni, K. et al. (2002). Bleu: a Method for Automatic Evaluation of Machine Translation. ACL:P02-1040.
• Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. ACL:W04-1013.
• Banerjee, S.; Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. ACL:W05-0909.
• Zhang, T. et al. (2019). BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675.
• Pillutla, K. et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. arXiv:2102.01454.
• Lin, S. et al. (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958.
• Parrish, A. et al. (2021). BBQ: A Hand-Built Bias Benchmark for Question Answering. arXiv:2110.08193.
• Dhamala, J. et al. (2021). BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation. arXiv:2101.11718.
• Hendrycks, D. et al. (2020). Measuring Massive Multitask Language Understanding. arXiv:2009.03300.
• Srivastava, A. et al. (2022). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv:2206.04615.
• Zhang, Z. et al. (2023). SafetyBench: Evaluating the Safety of Large Language Models. arXiv:2309.07045.
• Huang, H. et al. (2024). An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-Tuned Judge Model is not a General Substitute for GPT-4. arXiv:2403.02839.
• Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
• Gu, J. et al. (2024). A Survey on LLM-as-a-Judge. arXiv:2411.15594.
• Li, S. et al. (2025). LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge. arXiv:2506.09443.