Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs

What does this paper contribute to multimodal LLM evaluation?

The paper contributes a reproducible multimodal multiplication benchmark and a set of interpretability probes for measuring arithmetic failure in large language models. Unlike broad math benchmarks, this benchmark holds the underlying multiplication problem fixed while varying digit length, digit sparsity, representation, and modality.

• Controlled multimodal benchmark: The same multiplication instances are rendered as numeric text, number words, images, and audio where supported.
• Arithmetic-load measure: The paper defines C = total digits × non-zero digits as an interpretable proxy for operation count.
• Perception-versus-computation decomposition: Perception checks exceed 99% accuracy across modalities, while multiplication accuracy drops substantially.
• Heuristic fingerprinting: Forced-completion loss probes compare columnar multiplication, distributive decomposition, and rounding-compensation strategies.
• Adapter-geometry analysis: Heuristic-specific LoRA adapters reveal separated update directions but generally degrade arithmetic accuracy.

How is the multimodal multiplication benchmark designed?

The benchmark uses paired multiplication problems so that modality effects can be measured without changing the arithmetic task. It factorially varies digit length, non-zero digit count, representation, and modality to isolate whether errors come from perception, representation, or computation.

What is arithmetic load in this paper?

Arithmetic load is the paper’s compact difficulty measure for multiplication problems. It is defined as C = d_total × d_non-zero, where d_total is the total number of digits across both operands and d_non-zero is the number of non-zero digits.

Example: For 47 × 36, the total digit count is 4 and the non-zero digit count is 4, so C = 16. The paper reports that this simple load score remains predictive across modalities and models, with reported R² ≈ 0.50–0.67.

What are the key results across text and image inputs?

Accuracy falls as arithmetic load increases, and non-text inputs often degrade earlier than clean numeric text. On the held-out HDS probe split, Qwen3-VL-235B performs better than Qwen3-VL-30B, but both models show a measurable drop from text to image inputs.

Which multiplication strategy do the models prefer?

The forced-completion loss probe finds that distributive decomposition is the most compatible strategy for both Qwen3-VL model sizes. This preference appears in text and image inputs, although image presentation changes the margins between heuristic continuations.

A contrastive step probe further confirms that the models usually prefer correct intermediate arithmetic steps over matched incorrect ones. Correct-step preference is 100.0% for both models in text, 97.9% ± 1.2% for Qwen3-VL-30B in image, and 100.0% for Qwen3-VL-235B in image.

Do heuristic-specific LoRA adapters improve multiplication?

Heuristic-specific LoRA adapters change model behavior but usually do not improve arithmetic accuracy. Across held-out comparisons, degradations strongly outnumber improvements, suggesting that the base model’s native strategy routing is more reliable than forcing a single synthetic reasoning style.

The adapter-geometry analysis shows that heuristic updates are separated in parameter space. Effective-update cosine similarities are low: 0.0726–0.1247 for Qwen3-VL-30B and 0.0342–0.0586 for Qwen3-VL-235B across the DD, OT, and RC heuristic adapters.

Why does this contribution matter for multimodal AI systems?

The results show that exact arithmetic remains fragile even when multimodal models correctly perceive the input. This matters for real workflows involving screenshots, spreadsheets, slides, scanned documents, spoken prompts, and tool-using agents that must preserve mathematical structure before calling calculators or verifying results.

The benchmark separates three issues that are often entangled in math evaluation: recognizing the input, choosing a computation strategy, and executing exact arithmetic. That separation makes the benchmark useful for diagnosing whether future multimodal systems fail because they misread the problem or because they cannot reliably perform the computation.

Where is the multimodal multiplication benchmark available?

The benchmark data are available as paired text, image, and audio renderings of multiplication items. Access the MultimodalMathBenchmarks dataset on Hugging Face .

FAQ: Multimodal multiplication in LLMs