Authors: Samuel G. Balter | Ethan Jerzak | Connor T. Jerzak
Links: [PDF] [.bib]
Transcript
Introduction
Welcome to our talk on the ACL Findings paper, Multiplication in Multimodal LLMs. Let’s begin.
When an LLM sees 47 × 36, hears “forty-seven times thirty-six,” or reads that equation inside an image, we want the same thing: the exact product.
This project asks a simple question: does the modality or problem surface form change the computation?
The answer is yes, but in some subtle ways. Modality can shape how what we’ll call arithmetic load impacts performance and internal model routing.
Here is why that matters.
Problem Importance
Modern agents inspect screenshots, slides, scans, forms, and voice traces. In those settings, arithmetic errors are not cosmetic. A wrong product can become a wrong invoice, ledger entry, or downstream decision.
The key point is that difficulty is not just digit length. Sparse five-digit operands like 40,000 × 30,000 are structurally easier than dense five-digit operands like 45,327 × 36,215.
Same task, but different load.
Therefore, the benchmarks we’ll build must hold the arithmetic fixed while changing only the surface form and modality in these investigations
Benchmarks
Our benchmark starts from a generator. It creates one multiplication problem, then renders that same problem as numeric text, as an image, and, when possible, as audio.
That gives us 10,000 shared instances with one answer key. The comparisons here become clean: did the model fail because it could not read the input, or because it could not compute the product?
Without this pairing, modality effects get mixed with problem differences.
Now we need one scale for arithmetic difficulty.
Arithmetic Load, C
We will therefore define arithmetic load, C, as total digits times non-zero digits.
For 47 × 36, there are four total digits and four non-zero digits, so C equals 16.
C is not perfect. It ignores some carries and shortcuts. But it captures approximately how many digit interactions the model has to carry.
Notice the contrast at the bottom. 9999 × 8888 and 9000 × 8000 look similar in size, but their loads differ sharply, and alters the computational burden.
With C defined, the first result becomes visible.
Accuracy Cliff
Here is the first empirical result. Across six model families and variants, the probability of a correct answer falls as C increases.
The curves differ, but the performance cliff is consistent. Gemini 2.5 Flash, Qwen 30B, and GPT-4o cross 50 percent accuracy around C equals 50 to 54. Qwen 235B holds until roughly C equals 74 to 75.
GPT-5.4 and Grok look stronger in numeric settings, but become more fragile under alphabetic images.
So load is not the only thing. But it is the dominant axis.
Next, watch how the input channel shifts the cliff.
Modality Shifts
Numerical images often track numerical text closely. Qwen 235B, for example, is around C equals 74 in text and 75 in numerical images.
But alphabetic images expose a different weakness. GPT-5.4 drops about eight C-points from numeric to alphabetic image. Grok 4.20 drops about nineteen.
The same arithmetic can fail earlier when the model must translate a surface form into a calculation route.
That raises an obvious objection: maybe the model just misread the input.
Perception vs. Computation
Our perception robustness tests help answer that objection. On matched perception tasks, models are above 99 percent.
They can read the numbers. The failure appears later, when partial products, carries, and long-range digit interactions accumulate.
The bottleneck is the arithmetic process after perception.
So now we ask how the model tries to compute arithmetic here.
Heuristic Probe
The paper compares three familiar strategies.
OT is the columnar, schoolbook route. DD decomposes operands into place-value chunks. RC rounds to a nearby base and compensates.
Instead of trusting self-reports, we score short heuristic-specific continuations under forced completion. Lower loss means the model finds that continuation more natural in the same problem context.
This asks, “Which computational style does the model’s distribution prefer?”
The answer is surprisingly consistent.
Decomposition Preference
Across Qwen 30B and Qwen 235B, and across text and image, decomposition has the lowest forced-completion loss.
For 30B text, DD beats the neutral baseline by about 0.506 loss points. In image, DD is still the least costly heuristic.
For 235B, the heuristic prompts sit above neutral, but DD remains lowest loss.
So the model’s default approach is not columnar multiplication. It leans toward decomposition.
The natural intervention is to steer that heuristic choice. The result is cautionary.
LoRA Nudges
We train lightweight LoRA adapters for OT, DD, and RC traces. The hope is that steering the model toward the right trace will improve exact arithmetic.
It does not.
The adapters change behavior, but almost always hurt correctness. For Qwen 30B, 114 correctness flips include one improvement and 113 degradations. For Qwen 235B, 121 flips include four improvements and 117 degradations.
The lesson is not that strategies do not matter. The lesson here is that the base model’s router seems better than any single imposed heuristic.
Update Geometry
The LoRA updates for OT, DD, and RC move in nearly orthogonal directions.
Cross-heuristic cosine similarities are small: about 0.07 to 0.12 for 30B, and about 0.03 to 0.06 for 235B.
Same-heuristic reruns are more similar than cross-heuristic pairs. Strategy separation looks robust here. Forcing one route disrupts the learned router.
So the final message is not “pick one heuristic.” It is more practical than that.
Takeaways
Here is what to remember.
First, digit structure predicts failure better than modality labels alone.
Second, perception is not enough. Models can read the problem and produce inaccurate results.
Third, strategy priors are measurable. In these Qwen probes, decomposition is the dominant groove.
Fourth, rigid traces are dangerous. They can make arithmetic worse.
For more information about this and other projects, see neuristemic.ai
References
Samuel Balter, Ethan Jerzak, Connor T. Jerzak. Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs. Findings of the Association for Computational Linguistics (ACL), 2026
[PDF] [BibTeX]
[Overview] [Data] [>]