LLM Activation Patterns: Why Mathematical Reasoning Has the Highest Attention Entropy

Choosing which LLM to deploy in production is mostly vibes-based. You read benchmark leaderboards, pick the highest-scoring model that fits your budget, and ship. Most of those benchmarks tell you what the model can do, not how it processes the task internally.

I ran a study to look inside. The full code and data are at llm-neural-activation-patterns — a systematic comparison of how six different LLM architectures process twelve cognitive task categories, measured across 144 task-model combinations.

The headline result: mathematical reasoning consistently produces the highest attention entropy across every architecture tested — roughly 4x the entropy of scientific explanations. That's a structural fact about how transformers process math, not an artifact of one model.

The six architectures

I picked six LLMs spanning encoder, decoder, monolingual, multilingual, and efficiency-optimized designs, from 109M to 3.6B parameters: BERT-Base (109.5M, bidirectional encoder), GPT2-117M (124.4M, classic autoregressive decoder), Qwen-1.5-0.5B (464M, modern multilingual decoder), Phi-1 (1.4B, efficiency-optimized decoder), BLOOM-560M (559M, multilingual autoregressive), and StableLM-3B (3.6B, large-scale decoder).

This range matters. If you only measured GPT-family models, you'd miss the architectural differences. If you only measured 7B+ models, you'd miss how parameter scale interacts with task complexity.

The twelve cognitive tasks

Each model processed prompts from twelve categories: factual questions, creative writing, mathematical reasoning, emotional content, technical code, philosophical queries, conversational chat, logical puzzles, scientific explanations, language tasks, instruction following, and commonsense reasoning. The categorization spans retrieval, generation, reasoning, social cognition, and domain knowledge — not just easy vs hard.

The three metrics

Final Activation (Af) — mean magnitude of the final hidden layer. A proxy for processing intensity: how strongly the model is firing as it produces the answer.

Attention Entropy (Hatt) — Shannon entropy of attention weight distributions across all heads and layers. High entropy = attention spread across many tokens. Low entropy = focused attention.

Maximum Sparsity (Smax) — peak sparsity across network layers. Measures computational efficiency: how much of the network is silent at any given moment.

Finding 1: Mathematical reasoning has 4x the entropy of scientific explanations

Averaged across all six architectures: Mathematical Reasoning entropy 195.66 ± 46.66, Logical Puzzles 108.27 ± 44.75, Technical Code 94.11 ± 35.08, Scientific Explanations 47.03 ± 19.10.

This is the most counterintuitive result of the study. Naively, you'd expect scientific explanations to demand the most attention spread. The opposite is true.

Why: mathematical reasoning is fundamentally a multi-step process where intermediate state must be available across many tokens. The model can't just produce the next token from local context — it needs to refer back to assumptions, transformations, and partial results made earlier. That requires distributed attention. Scientific explanations, by contrast, are largely retrieval: the model has the answer stored in its weights, and produces it in a relatively localized way.

Practical implication: if you're deploying an LLM for math-heavy workloads (financial reasoning, scientific computing, code that involves quantitative logic), expect higher per-token compute and pay attention to context length. The model is doing more work per token than for harder-looking tasks like long-form explanations.

Finding 2: Decoder sparsity is 7x encoder sparsity

Decoder average max sparsity (5 models): 0.276. Encoder (BERT-Base): 0.039. Decoder models leave significantly more of their network silent during any given forward pass — a 7-fold difference. Decoder models are more amenable to inference-time efficiency tricks like pruning, mixture-of-experts routing, and sparse attention. Encoder models like BERT use more of their network per inference, which makes them harder to optimize for low-latency serving but means they're often more complete in their reasoning per token.

Finding 3: Parameter scaling is non-monotonic

The conventional wisdom is that bigger models always have stronger and more focused activations. The data says otherwise. Final activation values: BERT-Base (109.5M) -0.013, GPT2-117M (124.4M) 0.328, Qwen-1.5-0.5B (464M) -0.073, BLOOM-560M (559M) -1.836, Phi-1 (1.4B) 0.001, StableLM-3B (3.6B) 0.005.

GPT2-117M dominates final activation despite being the second-smallest model. BLOOM-560M produces the most negative activation despite being mid-size. Phi-1, optimized for efficiency, has remarkably low activation despite 1.4B parameters. When you scale up, you don't just get more of the same — architecture choices, training data composition, and optimization objectives produce qualitatively different activation regimes.

Finding 4: GPT2 dominates top-10 final activation positions

GPT2-117M occupies all top 10 positions for highest final activation magnitude across the 144 task-model combinations. It also has the lowest attention entropy of any decoder model (51.09). GPT2 is doing focused, intense processing on whatever it sees — likely an artifact of its training objective and corpus. Modern decoders trained with diverse objectives (instruction-following, RLHF, multilingual corpora) have broader, lower-intensity activation patterns. They've traded peak processing for breadth.

What this means if you're deploying LLMs in production

• If your task is math-heavy or multi-step reasoning, allocate more inference budget than for longer but more retrieval-oriented tasks. Per-token compute cost varies more than you'd expect from prompt length alone.

• If you're choosing between encoder and decoder architectures for a classification or embedding task, factor in the sparsity gap. Decoders are more amenable to inference-time efficiency techniques; encoders make fuller use of their network per call.

• Don't assume linear scaling in interpretability or activation behavior. A 3B-parameter model isn't behaving like a bigger GPT2. Architectural and training choices matter at least as much as parameter count.

Reproducing

git clone https://github.com/mahdinaser/llm-neural-activation-patterns.git
cd llm-neural-activation-patterns
pip install pandas openpyxl transformers torch
python scripts/brain.py

The full analysis tables (Tables II–VI in the paper) are in data/raw/. The Excel files contain all 144 task-model combinations if you want to run your own slicing.

Why I ran this study

Most LLM benchmarks treat the model as a black box: prompt in, score out. That gives you product-level insights but no architectural intuition for engineering decisions. For engineering teams making real deployment choices — which model to fine-tune, where to spend inference budget, when to compress, when to swap architectures — the inside view matters more than leaderboard rankings.

Code & data: github.com/mahdinaser/llm-neural-activation-patterns