TangoIA Benchmarks
Metrics
Categories
Themes
TangoIA Benchmarks is a Spanish-language evaluation leaderboard developed in Argentina for assessing the performance of large language models across a range of domains. It measures model accuracy and average scores on tasks spanning general knowledge, clinical, legal, news, humor, and mathematics categories.
The benchmark is designed specifically to address the need for Spanish-language evaluation tools, with a focus on Argentine linguistic and cultural context. It is intended for researchers and developers seeking to compare and assess AI model capabilities in Spanish, particularly within domains relevant to Latin American users and institutions.
TangoIA Benchmarks: Evaluating AI in the Latin American Context
The race to build capable regional language models has a new proving ground. Developed by Latin American AI infrastructure startup SuruS, the TangoIA Benchmarks serve as a definitive leaderboard for evaluating Large Language Models in Spanish. The benchmark specifically targets the Rio de la Plata dialect and the Argentine cultural context. Following the late 2024 release of Tango-70b, the first 70-billion parameter model trained for Latin America, the benchmark gained significant traction among researchers measuring how well open-weight models handle localized nuances.
What It Measures and Why It Matters
Most Spanish-language benchmarks rely on direct translations of English datasets or default to Iberian Spanish. TangoIA breaks this mold. It tests models on their grasp of regional idioms, local cultural references, and specific institutional knowledge relevant to Argentina and the broader Latin American market.
For machine learning engineers, this distinction matters. Generic Spanish models often fail at localized tasks. A model trained heavily on data from Spain might misinterpret Argentine legal text or fail to grasp regional humor. TangoIA provides a quantifiable metric for this cultural and linguistic alignment.
Evaluation Methodology and Key Metrics
The benchmark runs on a customized Hugging Face evaluation harness. It calculates an unweighted average across 23 distinct evaluation tasks, generating 46 total metric values. The methodology prioritizes reproducibility, allowing developers to run the evaluation script locally via a Python virtual environment.
The task categories are dense and varied. They include:
- Clinical: ClinDiagnosES and ClinTreatES test diagnostic and treatment reasoning in Spanish.
- News and Information: NoticIA and Fake News ES evaluate summarization and misinformation detection.
- Reasoning and Math: COPA_es for plausible alternative reasoning and MGSM_es for multilingual grade school math.
- Culture and Nuance: HumorQA and linguistic acceptability tests like EsCoLA.
Leaderboard Highlights and Evaluated Models
The current leaderboard highlights a massive performance gap between localized models and generalist open-weight models. SuruS's own Tango-70b currently dominates the benchmark with an average score of 59.90 percent. It excels particularly in reading comprehension and reasoning, scoring 92.00 on Belebele Spa and 89.60 on COPA_es.
Standard open models lag significantly when tested on these regional nuances. For example, Google's gemma-2-9b-it scored just 33.62 percent on the same 23 tasks, revealing a nearly 26-point deficit. The benchmark is now routinely used to evaluate fine-tunes of Llama 3.1, Mistral, and Gemma architectures targeting the Latin American market.
Limitations and Criticisms
Despite its utility, TangoIA faces valid structural criticisms. Its hyper-regional focus is a double-edged sword. While it perfectly captures Rio de la Plata Spanish, researchers note it does not accurately reflect model performance for Mexican, Colombian, or Iberian Spanish users.
This limitation directly spurred the July 2025 launch of La Leaderboard, a broader community-driven initiative that expanded on TangoIA's foundation to include 66 datasets covering multiple Spanish varieties and regional languages like Catalan and Basque. Additionally, some of TangoIA's underlying datasets rely on modified translations. Critics argue these datasets can occasionally introduce translation artifacts that skew evaluation results.
Even with these constraints, TangoIA remains a critical tool. It forces the AI community to look beyond generic language capabilities and measure the cultural competence of modern language models.