danish-foundation-models/multilingual-gsm-symbolic

Multilingual GSM-Symbolic

Multilingual GSM-Symbolic is a benchmark for evaluating arithmetic reasoning in large language models across multiple languages. It extends Apple's GSM-Symbolic approach by providing symbolic templates that generate thousands of structurally equivalent but numerically distinct math problems. Templates and generation are handled by the multilingual-gsm-symbolic package.

The dataset lets you test whether a model genuinely understands a problem or merely pattern-matches on the specific numbers it saw during training. Comparing *_original and *_synthetic splits directly measures this gap.

Want to add your own language to the dataset? It only requires validating 100 templates. Read more about how to contribute [here](https://github.com/centre-for-humanities-computing/multilingual-gsm-symbolic#want-to-add-a-new-language).

Dataset Structure

Each language is a separate config (subset). All languages provide test_original and test_synthetic; only English currently also provides train_original and train_synthetic.

Split	Description
test_original	The 100 concrete GSM problems for that language
test_synthetic	20 generated variants per template (2,000 problems)
train_original	English only: 7.47k concrete training problems
train_synthetic	English only: 20 generated variants per template (149k problems)

Fields

Field	Type	Description
question	string	The math problem
answer	string	Step-by-step solution ending with #### <number>
target	string	The final numeric answer (extracted from answer)
language	string	Three-letter language code (e.g. eng, dan)
source_id	int	Problem index in the original GSM8K dataset

Answer format

Answers follow the GSM8K convention — reasoning steps followed by a final numeric answer:

At 3 miles/hour, it will take 42/3=14 hours for the fog to cover the city.
#### 14

Loading the Dataset

With 🤗 datasets

from datasets import load_dataset

# English synthetic test split
eng = load_dataset("danish-foundation-models/multilingual-gsm-symbolic", name="eng", split="test_synthetic")

# German original test split
deu = load_dataset("danish-foundation-models/multilingual-gsm-symbolic", name="deu", split="test_original")

With inspect-ai

You can evaluate with inspect-ai simply using:

inspect eval hf/danish-foundation-models/multilingual-gsm-symbolic/synthetic_eng --model openai/gpt-5.4-nano --reasoning-effort low

This uses the eval.yaml in the repository. You can target any task id (e.g. original_deu, synthetic_isl) or omit the task name to run all languages.

Evaluation Results

As a sanity check for the results we evaluated the model with inspect-ai using openai/gpt-5.4-nano (reasoning effort: low) on Danish and English.

Language	Original accuracy	Synthetic accuracy
English	90.0%	75.2%
Danish	83.2%	70.2%

The gap between original and synthetic accuracy reflects performance degradation on novel number combinations — a proxy for how much a model relies on memorisation vs. genuine reasoning.

Performance degradation (original → synthetic)

Language gap (English vs. Danish, synthetic)

Accuracy by problem difficulty (reasoning steps)

Eval logs are available in the logs/ folder on this repository.

Citation

If you use this dataset, please cite the project in which it was constructed (Danish Foundation Models) as well as the original GSM-Symbolic paper:

@misc{enevoldsen2023danishfoundationmodels,
      title={Danish Foundation Models}, 
      author={Kenneth Enevoldsen and Lasse Hansen and Dan S. Nielsen and Rasmus A. F. Egebæk and Søren V. Holm and Martin C. Nielsen and Martin Bernstorff and Rasmus Larsen and Peter B. Jørgensen and Malte Højmark-Bertelsen and Peter B. Vahlstrup and Per Møldrup-Dalum and Kristoffer Nielbo},
      year={2023},
      eprint={2311.07264},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2311.07264}, 
}

@article{mirzadeh2024gsmsymbolic,
  title={GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models},
  author={Mirzadeh, Iman and others},
  year={2024}
}

Acknowledgements

This dataset was developed at the Centre for Humanities Computing by Kenneth Enevoldsen using the multilingual-gsm-symbolic package.