Metrics
Categories
Themes
Compar:IA is an open French-language platform developed by the French Ministry of Culture's digital service in partnership with PEReN (Pôle d'Expertise de la Régulation Numérique) for blind pairwise comparison of conversational AI models. Users interact with two anonymous models simultaneously and vote on their preferred response, with results aggregated using Bradley-Terry scoring alongside satisfaction scores to rank models.
The platform currently covers 82 evaluated models and is designed to collect user preference data that feeds into openly licensed French-language alignment datasets published on Hugging Face. It is notably one of the few government-led AI evaluation initiatives, making it relevant to researchers, policymakers, and developers interested in French-language model performance and public-sector approaches to AI benchmarking.
Background and Development
Compar:IA is a French public-sector initiative developed by the Service du numérique of the Ministère de la Culture (French Ministry of Culture) in collaboration with PEReN (Pôle d'Expertise de la Régulation Numérique), a French government body specializing in digital regulation expertise. The platform was created to address a specific gap in AI evaluation: the relative scarcity of openly available, French-language preference data for conversational AI models. It operates under an open license, with datasets published on Hugging Face under the ministere-culture organization profile.
The project is hosted at comparia.beta.gouv.fr, situating it within the broader beta.gouv.fr ecosystem of French public digital services. Its government origin makes it a distinctive example of state-led AI benchmarking infrastructure, positioned outside the commercial AI sector.
How the Platform Works
Compar:IA uses a blind pairwise comparison methodology. Users submit a prompt and receive simultaneous responses from two anonymized conversational AI models. After reviewing both outputs, the user votes for their preferred response. Model identities are revealed only after a vote is cast, reducing the influence of brand recognition or prior model familiarity on user judgments.
Votes are aggregated using Bradley-Terry scoring, a statistical method commonly used in comparative ranking systems to estimate the relative strength of competitors based on pairwise outcomes. Satisfaction scores are collected alongside preference votes as a secondary metric. As of the available data, the leaderboard covers 82 evaluated models, spanning a range of conversational AI systems assessed through this community-driven process.
Dataset Contributions and Openness
A central objective of Compar:IA is the creation of open French-language alignment datasets derived from user interaction and preference data. These datasets are made publicly available on Hugging Face, intended to support researchers and developers working on French-language model alignment, fine-tuning, and evaluation. The open licensing approach contrasts with many proprietary evaluation efforts where preference data remains internal to the collecting organization.
The platform's focus on French specifically addresses the historically underrepresented status of French in large-scale human preference datasets, which have tended to concentrate on English-language interactions.
Relevance and Use Cases
Compar:IA serves several distinct audiences:
- Researchers interested in human preference data, French-language NLP, or comparative evaluation methodologies can access the openly published datasets and leaderboard results.
- Policymakers and regulators may find the platform relevant as an example of government-led AI transparency and evaluation infrastructure.
- Model developers targeting French-language performance can use the leaderboard as an independent reference point for how their models compare to others under real user interaction conditions.
- General public users contribute to the evaluation process directly through participation, making the benchmark crowd-sourced in nature.
As a government-operated, open-source leaderboard focused on a non-English language, Compar:IA occupies a relatively uncommon position in the broader AI benchmarking landscape, which is dominated by English-language and commercially operated evaluation frameworks.