A Compositional Analysis of Cross-Lingual Math Reasoning in Efficient Vision-Language Models

Abstract. We evaluate two 4B-parameter vision-language models (VLMs) on grade-school math in five languages, independently varying the language of the question, its modality (text or rendered image), and the language the model reasons in. Cross-lingual accuracy is compositional: a two-term linear model in the model's monolingual accuracy on source-language images and its monolingual accuracy on target-language text predicts held-out cross-lingual accuracy with LOOCV $R^2 = 0.875$. Reading the question from the image accounts for most of the error; the choice of reasoning language and prompt strategy accounts for little.

1. Introduction

Small VLMs run on-device, where cloud models are expensive or unavailable. A student who photographs a worksheet and asks for help poses a task with two steps: the model reads the question, which may be in any language and may arrive as pixels, and then reasons to an answer, possibly in another language. Multilingual benchmarks report one accuracy for the whole task, so when a model fails, the score cannot say whether it misread the question or reasoned badly.

We measure the two steps separately. We take the 250 MGSM test questions in five languages, present each as plain text and as a rendered image, and ask the model to reason in each of the five languages in turn. Every combination of question language, modality, and reasoning language is its own condition, so a model's monolingual scores (reading and reasoning in the same language) can be compared against its cross-lingual scores (reading in one language, reasoning in another).

Cross-lingual visual reasoning in small VLMs decomposes into reading the source language from pixels and reasoning in the target language as text, and the model's monolingual scores on each step predict its cross-lingual accuracy. Three further findings follow from the same grid: reading limits accuracy far more than reasoning does, English is a robust reasoning language for non-English inputs, and asking the model to translate first does not help.

2. Setup

Models. Gemma 4 E4B and Qwen3.5 4B, 4-bit quantized, run locally with MLX on Apple Silicon. Inference only, greedy decoding.

Data. The 250-question MGSM test set in English, German, Spanish, Chinese, and Turkish. MGSM has no Turkish split, so we translated the full test set; we release the translation with the evaluation code.

Modalities. Each question appears as a text string and as a PNG rendered at 300 DPI (Fig. 1).

Prompt strategies. Cross-lingual conditions use three prompts. Original asks the model to reason in the target language and leaves the transfer implicit. Translate asks it to translate the question first, then solve it. No-Translate asks it to solve the question directly and forbids translation.

Notation. A cell is one model, one data language $d$ (the language of the question), one reasoning language $c$, one modality, and one prompt strategy. $A_t(d, c)$ is accuracy on text input and $A_v(d, c)$ is accuracy on image input. Cells with $d = c$ are monolingual; cells with $d \neq c$ are cross-lingual, and we call them off-diagonal. Five languages give 20 off-diagonal pairs, so each modality has 120 off-diagonal cells across both models and all three strategies (20 pairs × 2 models × 3 strategies).

The full grid is 5 data languages × 5 reasoning languages × 2 models × 3 strategies × 2 modalities: 300 conditions and 75,000 question-level evaluations.

3. Results

3.1 Reading limits accuracy

Average accuracy over all language pairs:

Image input costs 12.6 points on Gemma and 5.0 on Qwen, and the data language drives most of that loss. On Gemma, English images average 86.8% and Chinese images 53.2%, a 33.6-point gap. For a fixed data language, changing the reasoning language moves accuracy by at most 14 points. The bottleneck is getting the question out of the pixels.

3.2 Monolingual scores predict cross-lingual accuracy

For each off-diagonal cell we ask whether its accuracy can be predicted from the model's monolingual scores alone. We fit a linear model with two predictors, one for the source side and one for the target side, and score the fit by leave-one-out cross-validation (LOOCV). For text input, the predictors are the model's monolingual text accuracy in the data language and in the reasoning language:

For image input, the source-side predictor becomes monolingual image accuracy and the target-side predictor stays text:

This model fits all 120 off-diagonal image cells (Fig. 2). Replacing the target-side predictor with monolingual image accuracy $A_v(c, c)$ lowers the fit. Once the model has read the question off the image, reasoning in the target language is a text task.

3.3 English is a robust reasoning language

For non-English inputs, English reasoning is often the best or tied-best choice, and the tradeoff is asymmetric. Where native-language reasoning wins, it wins by at most 2.8 points; where English wins, the margin reaches 9.2 points (Gemma on Chinese images). A deployment restricted to one reasoning language should use a high-resource one.

3.4 Explicit translation does not help

Translate-then-solve lowers image accuracy on Gemma from 73.5% to 71.3% and has inconsistent effects on text. Leaving the transfer implicit is the simplest effective choice. Across the grid, prompt strategy moves accuracy by 2 to 3 points, an order of magnitude less than the data language does. Prompting cannot supply a capability the model lacks.

4. Robustness

Gemma's Chinese-image cells form a low outlying cluster (Fig. 2), and a regression across two clusters can report a high $R^2$ while explaining little within either. Two checks address this. Adding per-language and per-model group controls raises the target-reasoning coefficient $\beta_2$ from 0.61 to 0.67. Removing every Gemma × Chinese cell leaves both predictors highly significant, with $R^2 = 0.79$. The compositional structure survives without its worst cluster.

5. Limitations

We study two models, both 4B-parameter, 4-bit quantized, and decoded greedily; larger or full-precision models may compose differently. The task is grade-school math, where reading and reasoning separate cleanly, over five languages and 250 questions per language.

6. Conclusion

In small VLMs, reading the source language from pixels and reasoning in the target language as text are separable capabilities, and training coverage in one does not substitute for the other. A model that solves Turkish math problems from worksheet photos needs Turkish visual-language data and strong text reasoning in some language. Prompt strategy closes neither gap.

Data and code. The Turkish translation of the MGSM test set and all evaluation code are released with the paper.