Monthly · Open methodology · 5 dialects

Which AI agents actually work in Arabic?

أي النماذج تستدعي الأدوات بشكل صحيح باللهجة الخليجية والعربية؟

An independent monthly leaderboard of LLM function calling in Arabic, scored per dialect on an open benchmark.

5Dialects

27Real-world tools

8Domains

MonthlyRe-run cadence

Leaderboardلوحة الصدارة

First run: scheduled

◷

First results coming soon

We're running the first evaluation now. The greyed rows below are examples of the layout, not scores.

Swipe for all dialects →

#	Modelالنموذج	Tool accuracyدقة الأداة	Args completeاكتمال الوسائط	Gulfخليجي	Egyptianمصري	Levantineشامي	Maghrebiمغاربي	MSAفصحى

Scores are % correct on a fixed sample of AISA-ArabicFC. Click a column to sort.How we score →

Single run · temperature 0 · 246-item sample. Ranks within ~2–3 points of each other are within noise.

Coming soon: HUMAIN M3 (Saudi) and Fanar (Qatar)

01 · SAMPLE

A fixed sample from AISA-ArabicFC, balanced across five dialects, including requests that need no tool at all.

02 · RUN

Each model gets identical tool schemas through one OpenAI-compatible API. Temperature 0, one answer per item (network errors are retried).

03 · SCORE

We check that the model picked the right tool, then that every required argument is filled in, per dialect.