Open methodology

Methodology

المنهجية

Data

We draw a fixed-seed sample from the dev split of AISA-ArabicFC, with up to 60 items from each of MSA, Gulf, Egyptian and Levantine. The dev split has only 6 Maghrebi items, so all 6 are included and that column is marked low-sample. It includes requests where no tool should be called. A second suite, arabic-agent-eval (Apache-2.0), is planned.

Scoring

• Tool accuracy: the model picked the correct tool, or correctly picked none.
• Args complete: correct tool and every argument the reference call uses is filled in. We don't require exact string matches, because a valid answer can format values differently (e.g. "2025-02-01" vs "1 فبراير", "en" vs "English").
• Per-dialect: tool accuracy within each dialect.

Caveats

This is a sample, not the full benchmark or the hidden shared-task test set. These are not official AISA results. The eval script is open source and reproducible with run_eval.py --seed 0.