FLORES-200

From Systems Analysis Wiki
Jump to navigation Jump to search

FLORES-200 is an evaluation dataset for multilingual machine translation, covering approximately 200 world languages. It was created by researchers at Meta as part of the No Language Left Behind (NLLB) project and was introduced in 2022. FLORES-200 is an extension of the previous FLORES-101 dataset and is designed for the objective evaluation of translation quality, especially for low-resource languages[1].

A key feature of the dataset is that all texts were translated by professional translators, which ensures high-quality reference translations and makes it a reliable benchmark for comparing machine translation systems[2].

Background and Creation

The first version, FLORES-101, was introduced in 2022 and contained 3001 sentences from the English Wikipedia, translated into 101 languages[3]. This dataset filled an important gap in the evaluation of translation systems for languages with limited data.

In 2022, as part of the No Language Left Behind project, Meta expanded the corpus to 200 languages, creating FLORES-200[4]. The development faced several challenges:

  • Many of the added languages had low standardization and a lack of bilingual specialists.
  • Some languages were translated not directly from English but through pivot languages (Spanish, French, Russian).
  • For some languages, different writing systems (e.g., Latin and Cyrillic) were included to account for their use in different communities[5].

Composition and Structure

The FLORES-200 corpus includes 3001 sentences selected from 842 different web articles and documents from Wikimedia projects. Unlike the first version, the sources include not only Wikipedia but also other projects such as Wikinews, Wikijunior, and Wikivoyage. This ensures broad thematic diversity (news, science, culture, travel), allowing for a comprehensive evaluation of translation quality.

Each English sentence was professionally translated into ~200 target languages, forming a fully aligned parallel corpus. The dataset is divided into three parts:

  • dev (development) — for fine-tuning models.
  • devtest (intermediate testing) — for preliminary evaluation.
  • test (final testing) — a hidden part for fair comparison of models in competitions.

Languages are identified using the ISO 639-3 standard with a script identifier, for example, `eng_Latn` for English in Latin script or `rus_Cyrl` for Russian in Cyrillic script[5].

Application and Significance

FLORES-200 has become a key benchmark for evaluating multilingual machine translation systems. It was used to evaluate Meta's flagship model, NLLB-200. Testing on FLORES-200 showed that NLLB-200 improved translation quality by an average of 44% on the BLEU metric compared to previous state-of-the-art systems[6]. For some African and Indian languages, the accuracy improvement exceeded 70%[4].

Meta has made the dataset and its tools freely available under the Creative Commons BY-SA 4.0 license. As a result, FLORES-200 has been widely adopted and has become a de facto standard in academic research, machine translation competitions (such as WMT), and language preservation initiatives. In 2023, the OLDI (Open Language Data Initiative) community began expanding the corpus under the name FLORES+[2].

See also

Literature

  • Liang, P. et al. (2022). Holistic Evaluation of Language Models (HELM). arXiv:2211.09110.
  • Chang, Y. et al. (2023). A Survey on Evaluation of Large Language Models. arXiv:2307.03109.
  • Ni, S. et al. (2025). A Survey on Large Language Model Benchmarks. arXiv:2508.15361.
  • Biderman, S. et al. (2024). The Language Model Evaluation Harness (lm-eval): Guidance and Lessons Learned. arXiv:2405.14782.
  • Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. arXiv:2104.14337.
  • Ma, Z. et al. (2021). Dynaboard: An Evaluation‑As‑A‑Service Platform for Holistic Next‑Generation Benchmarking. arXiv:2106.06052.
  • Goel, K. et al. (2021). Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv:2101.04840.
  • Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models: A Survey. arXiv:2406.04244.
  • Liu, S. et al. (2025). A Comprehensive Survey on Safety Evaluation of LLMs. arXiv:2506.11094.
  • Chiang, W.-L. et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.
  • Boubdir, M. et al. (2023). Elo Uncovered: Robustness and Best Practices in Language Model Evaluation. arXiv:2311.17295.
  • Huang, L. et al. (2023). A Survey on Hallucination in Large Language Models. arXiv:2311.05232.

References

  1. ↑ “FLoRes-200 Dataset”. Papers With Code. [1]
  2. ↑ 2.0 2.1 “FLORES+ Translation and Machine Translation Evaluation for the Erzya Language”. Proceedings of the Ninth Conference on Machine Translation (WMT24). [2]
  3. ↑ Goyal, N., et al. “The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation”. Transactions of the Association for Computational Linguistics. [3]
  4. ↑ 4.0 4.1 “New AI Model Translates 200 Languages, Making Technology Accessible to More People”. Meta Newsroom. [4]
  5. ↑ 5.0 5.1 “Muennighoff/flores200”. Hugging Face. [5]
  6. ↑ Costa-jussà, M.R., et al. “No Language Left Behind: Scaling Human-Centered Machine Translation”. arXiv:2207.04672. [6]
Retrieved from "https://systems-analysis.ru/eng/index.php?title=FLORES-200&oldid=2679"