BGPA

The Benchmark Stress-Testing LLMs on Brazilian Accounting

Evaluating large language models on general knowledge is a solved problem. Evaluating them on highly localized, domain-specific professional standards is not. Enter the Brazilian Graduate Proficiency in Accounting (BGPA) benchmark.

Created in 2025 by Roberto Dias Duarte, BGPA is a text-based evaluation suite designed specifically for the Brazilian financial context. It measures how well LLMs comprehend and reason through complex accounting and finance principles in Portuguese. For machine learning engineers building financial AI agents, BGPA has become a critical tool for separating models that actually understand Brazilian regulations from those that merely translate US-centric financial data.

What It Measures and Why It Matters

Brazil operates under a notoriously intricate tax and regulatory framework. Generalist models frequently fail here. They hallucinate tax codes, confuse US GAAP with Brazilian CPC (Comitê de Pronunciamentos Contábeis) standards, or stumble over localized financial terminology.

BGPA targets these exact blind spots. It forces models to navigate the specific regulatory and educational environment of Brazil. For researchers, it matters because it tests a model's localized knowledge retrieval and domain-specific reasoning, rather than its general linguistic competence. High performance on BGPA indicates an LLM is viable for downstream tasks like automated auditing, tax classification, and financial advisory in South America's largest economy.

Evaluation Methodology

The BGPA dataset is lean but rigorous. It consists of 200 multiple-choice questions extracted directly from graduate-level accounting subject matter. The methodology strips away conversational fluff to focus purely on technical accuracy.

• Dataset Size: 200 graduate-level questions.
• Language: Brazilian Portuguese.
• Primary Metric: Accuracy (percentage of correct answers).
• Format: Multiple-choice, requiring definitive answers without generative hedging.

Engineers typically run these evaluations in zero-shot or few-shot settings to measure the model's base internalization of accounting principles without extensive prompt engineering.

Leaderboard Highlights: Testing 21 Models

Since its launch, researchers have run 21 different LLMs through the BGPA gauntlet. The results highlight a persistent gap in localized training data.

Massive proprietary models from labs like OpenAI, Google, and Anthropic generally lead the pack due to their vast parameter counts and superior cross-lingual reasoning. However, the benchmark exposes significant weaknesses in mid-tier and open-weights models when forced to reason about Brazilian-specific financial scenarios. Notably, models fine-tuned specifically on Portuguese corpora often punch above their weight class on the BGPA leaderboard. This proves that targeted, high-quality local data can rival sheer compute scale in specialized professional domains.

Limitations and Known Criticisms

Despite its utility, BGPA faces valid structural criticisms from the ML community.

First, the dataset size is small. At just 200 questions, the benchmark is highly susceptible to data contamination. If a model's training pipeline ingests the specific graduate exams used to build BGPA, its accuracy score will artificially inflate, rendering the evaluation useless.

Second, multiple-choice accuracy does not perfectly map to real-world utility. An LLM might select the correct regulatory answer from a list of four, but still fail to draft a coherent, compliant financial report.

Finally, Brazilian tax and accounting laws are highly dynamic. A static benchmark created in 2025 risks rapid deprecation. To remain a definitive standard through 2026 and beyond, BGPA will require continuous versioning to reflect the latest legislative updates.