Open Arabic LLM Leaderboard

Background and Development

The Open Arabic LLM Leaderboard (OALL) was created in 2024 by the Technology Innovation Institute (TII), a research organization based in the United Arab Emirates. Its development responded to a recognized gap in the Arabic natural language processing (NLP) community: the absence of a standardized, publicly accessible platform for systematically evaluating Arabic large language models. As Arabic AI research has grown, the need for consistent evaluation infrastructure has become increasingly apparent, and OALL was designed to address that need directly.

The leaderboard is hosted on HuggingFace Spaces and built on HuggingFace's LightEval framework, a modular evaluation library that supports reproducible and transparent model assessments. This technical foundation allows researchers and developers to submit models and receive comparable results across a shared set of benchmarks.

Benchmark Coverage and Evaluation Tasks

OALL aggregates four established Arabic-language benchmarks to assess model capabilities across a range of tasks and domains:

• AlGhafa: A benchmark covering multiple Arabic NLP tasks, developed to assess general Arabic language understanding.
• ACVA (Arabic Cultural Values and Alignment): Focuses on culturally grounded question answering relevant to Arabic-speaking contexts.
• Arabic MMLU: An Arabic adaptation of the Massive Multitask Language Understanding benchmark, covering academic and professional subject knowledge in multiple-choice format.
• Arabic EXAMS: A benchmark derived from Arabic examination materials, testing subject-specific reasoning and comprehension.

Together, these benchmarks span tasks including reading comprehension, sentiment analysis, question answering, and multiple-choice evaluation, offering a broad view of a model's linguistic and reasoning capabilities in Arabic.

Use Cases and Target Audience

OALL is primarily intended for researchers and developers working on Arabic NLP and large language model development. It provides a centralized location for comparing how different models perform on standardized Arabic tasks, which can inform decisions about model selection, fine-tuning strategies, and research priorities. Academic institutions, AI laboratories, and industry teams developing Arabic-language applications can use the leaderboard to benchmark their models against others in the field.

By making results publicly visible on HuggingFace Spaces, OALL also contributes to broader transparency in Arabic AI evaluation, allowing the community to track progress over time and identify areas where model performance remains limited.

Significance for Arabic NLP Research

Arabic presents particular challenges for language model evaluation, including morphological complexity, dialectal variation, and a relative scarcity of standardized benchmarking tools compared to English-language resources. OALL addresses part of this gap by providing a unified evaluation platform that draws on multiple benchmarks rather than relying on a single dataset, offering a more comprehensive picture of model performance.

The leaderboard's integration with the HuggingFace ecosystem lowers barriers to participation, as researchers already familiar with that platform can submit and evaluate models without requiring separate infrastructure. As Arabic AI research continues to expand, platforms like OALL contribute to establishing shared standards that enable more consistent comparison across models and research groups.