# CMSC 25750/35750: ## Large Language Models ## Lecture 3: MLPs and Factual Recall <p style="text-align: center;"> Chenhao Tan<br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p>
## Logistics - Roast or toast signup - Project submissions - Office hour information online - Any feedback
## From attention to MLPs Last lecture: attention heads move information between token positions. But feed-forward layers (MLPs) account for two-thirds of a transformer's parameters. What do they do? This lecture follows a two-paper arc: - Meng et al. (2022): mid-layer MLPs store factual associations, which can be directly edited (ROME) - Niu et al. (2024): the same localization methods work on syntax — so what are MLPs actually storing?
## Today's agenda * Paper overviews * Deep dives * Discussion
## Today's agenda * Paper overviews * <span style="color: #767676;">Deep dives</span> * <span style="color: #767676;">Discussion</span>
## Locating and Editing Factual Associations in GPT Meng et al. (2022) use causal tracing to find where factual associations are stored. Finding: mid-layer MLP modules at the last subject token are decisive. A separate late site — attention at the last token — copies the recalled fact to the output. They then introduce ROME (Rank-One Model Editing): directly update the MLP weights at the decisive layer to insert a new factual association. <p class="citation"><a href="https://arxiv.org/abs/2202.05262">Meng et al., 2022</a></p>
## What does the Knowledge Neuron Thesis Have to do with Knowledge? Niu et al. (2024) ask: if the same methods that localize facts also localize syntactic phenomena (number agreement, subject-verb agreement), do MLPs actually store "knowledge"? They find that KN-edit has low reliability (max ~5% overturn rate for syntax, up to 48% for facts). ROME fails under symmetry and synonymy tests. Their conclusion: MLPs store token expression patterns, not knowledge in any meaningful sense. <p class="citation"><a href="https://arxiv.org/abs/2405.02421">Niu et al., 2024</a></p>
## The arc | | Meng et al. | Niu et al. | |---|---|---| | Question | Where are facts stored? | Is that really knowledge? | | Method | Causal tracing + model editing | Syntactic probing + editing eval | | Claim | Mid-layer MLPs store facts | MLPs store patterns, not knowledge |
## Today's agenda * <span style="color: #767676;">Paper overviews</span> * Deep dives * <span style="color: #767676;">Discussion</span>
## Causal tracing Meng et al. run three forward passes to locate factual recall: 1. Clean run: normal forward pass on "The Space Needle is in downtown ___" — model predicts "Seattle" 2. Corrupted run: add Gaussian noise to subject token embeddings — model loses the subject, likely predicts incorrectly 3. Restore run: run on corrupted input, but at one specific (layer, token) position restore the clean hidden state — measure if the correct prediction recovers The indirect effect (IE) of a state is the probability recovery it causes while the subject remains corrupted. <p class="citation">Meng et al., 2022</p>
## Why "indirect" effect? From causal mediation analysis: the total effect of a cause on an output splits into a direct path and paths through mediators. In causal tracing, the subject remains corrupted throughout, so all states computed from it are also corrupted. Restoring one hidden state \\(h^{(l)}_t\\) to its clean value asks: how much of the correct prediction flows forward from this node alone? This is the indirect effect mediated by \\(h^{(l)}_t\\) — the portion of the total treatment effect (clean vs. corrupted subject) that is channeled through this specific state. <p class="citation">Meng et al., 2022; Wang et al., 2022</p>
## Two sites of factual recall Averaging over 1000 factual statements in GPT-2 XL: Early site: mid-layer MLP modules at the last subject token have high average indirect effect (AIE up to 6.6% for MLPs vs 1.6% for attention). Late site: attention at the last token of the prompt (just before the prediction) has strong causal effect — this is where the recalled fact is copied to the output position. The hypothesis: mid-layer MLPs recall the fact about the subject; late-layer attention copies it to the last token for prediction. <p class="citation">Meng et al., 2022</p>
## Causal tracing vs. the IOI approach Both use causal interventions, but differ in direction and granularity. IOI (Wang et al.): start from a clean run, patch in corrupted activations for one component at a time, measure the **drop** in logit difference. Identifies components that are necessary — if removing it hurts, it matters. Causal tracing (Meng et al.): start from a fully corrupted run, restore one hidden state at a time, measure **recovery** of correct probability. Identifies states that are sufficient to carry the signal forward. <p class="citation">Meng et al., 2022</p>
## Causal tracing vs. path patching | | Causal tracing (Meng et al.) | Path patching (Wang et al.) | |---|---|---| | Direction | Recovery: corrupt all, restore one | Drop: clean baseline, corrupt one | | What it finds | States sufficient to relay the signal | Components necessary for the output | | Pro | Pinpoints where signal can be recovered | Identifies load-bearing components | | Con | May highlight relay nodes, not storage sites | Misses redundant components (backup paths compensate) | <p class="citation">Meng et al., 2022; Wang et al., 2022</p>
## ROME: overview Goal: change what the model says about a subject — e.g., replace "Space Needle is in Seattle" with "Space Needle is in Paris" — by directly modifying weights, not by fine-tuning. Three steps: 1. Use causal tracing to identify the decisive MLP layer \\(l^\ast\\) at the last subject token 2. Compute a key \\(k_\ast\\) (what the subject looks like at \\(l^\ast\\)) and a value \\(v_\ast\\) (what the output should be) 3. Update \\(W_{\text{proj}}^{(l^\ast)}\\) with a rank-one edit that inserts \\((k_\ast, v_\ast)\\) while preserving existing associations The key insight: \\(W_{\text{proj}}\\) can be treated as a key-value memory, so inserting one new association is a well-defined algebraic problem. <p class="citation">Meng et al., 2022</p>
## ROME: \\(W_{\text{proj}}\\) as associative memory A transformer MLP has two weight matrices. The first (\\(W_{fc}\\)) projects up and detects patterns; the second (\\(W_{proj}\\)) projects back down and produces the output contribution. \\(W_\text{proj}^{(l)}\\) has shape \\(d_\text{model} \times d_\text{ff}\\). Each column is a "value vector" that gets activated when the corresponding "key" (row of \\(W_{fc}\\)) fires. Meng et al. model this as a linear associative memory: given a set of key vectors \\(K\\) and desired value vectors \\(V\\), the matrix \\(W\\) encodes the associations by satisfying \\(WK \approx V\\). <p class="citation">Meng et al., 2022</p>
h
d_model
W_fc
keys: detect patterns
σ(W_fc h)
d_ff = 4 × d_model
W_proj
values: produce output
m
d_model
## ROME: \\(k_\ast\\) and \\(v_\ast\\) Key \\(k_\ast\\): the hidden state at layer \\(l^\ast\\) when the model processes the subject. Averaged over a small set of prompts ending with the subject, so it represents the subject stably rather than one surface form. Value \\(v_\ast\\): optimized by gradient descent so that, when the model reads \\(k_\ast\\) at layer \\(l^\ast\\), it produces the desired output. This encodes the new relation \\((r, o^\ast)\\) as a property of the subject. <p class="citation">Meng et al., 2022</p>
## ROME: the rank-one update To insert \\((k_\ast, v_\ast)\\) while preserving existing associations \\(WK \approx V\\), they solve: \\[\hat{W} = W + \Lambda (C^{-1} k_\ast)^\top\\] \\(C\\) is the key covariance, pre-computed once. \\(C^{-1} k_\ast\\) downweights directions that are frequently activated, concentrating the edit where it causes the least interference. <p class="citation">Meng et al., 2022</p>
## ROME: solving for \\(\Lambda\\) Set \\(\Lambda = \frac{v_\ast - W k_\ast}{k_\ast^\top C^{-1} k_\ast}\\) and verify \\(\hat{W} k_\ast = v_\ast\\): \\[\hat{W} k_\ast = W k_\ast + \Lambda\,(C^{-1} k_\ast)^\top k_\ast = W k_\ast + \frac{v_\ast - W k_\ast}{k_\ast^\top C^{-1} k_\ast} \cdot k_\ast^\top C^{-1} k_\ast = v_\ast\\] The \\(C^{-1}\\) cancels exactly: \\((C^{-1} k_\ast)^\top k_\ast = k_\ast^\top C^{-1} k_\ast\\) appears in both numerator and denominator. The residual \\(v_\ast - W k_\ast\\) is what the current weights are missing. \\(\Lambda\\) is solved analytically — no optimization needed at this step. <p class="citation">Meng et al., 2022</p>
## Tightening the evaluation criteria Meng et al. evaluate on reliability (does the edit work?), generality (does it transfer to paraphrases?), and locality (does it leave unrelated facts unchanged?). Yao et al. (2023) add portability: if you change Watts Humphrey's alma mater to Harvard, the model should also know that Harvard is in Boston. Portability drops to ~50% where generality is ~90%. Niu et al. further add symmetry and synonymy: a truly successful edit should propagate to logically equivalent reformulations. Each new criterion reveals a gap that the previous ones missed. <p class="citation">Niu et al., 2024</p>
## Localizing syntax the same way Niu et al. apply the same KN-edit localization method (integral of gradients) to syntactic phenomena from BLiMP: determiner-noun agreement, subject-verb agreement, anaphoric gender agreement. Finding: syntactic phenomena localize to just 2-5 neurons, just like factual associations. The localization is highly consistent across prompts. Furthermore, syntactic and factual neurons occupy the same layers (the topmost layers), with no clear bifurcation — contradicting the view that syntax is shallow and semantics is deep. <p class="citation">Niu et al., 2024</p>
## Formal vs. functional competence Mahowald et al. (2024) distinguish two types of competence: - Formal: knowledge of linguistic rules and patterns (syntax) - Functional: understanding and using language in the world (semantics, facts) The KN thesis implicitly assumes functional competence is special — that "knowledge" is stored differently from rule-following. Niu et al.'s finding that both localize the same way, in the same layers, using the same method, challenges this assumption. If the mechanism is the same, the distinction may not be meaningful at the level of internal representations. <p class="citation">Niu et al., 2024</p>
## KN edit has low reliability The core test of the KN thesis: can suppressing the identified neurons overturn the model's categorical prediction? For determiner-noun agreement: only 5.2% of predictions are overturned. For factual editing: reliability ranges from 1.66% to 47.86% across benchmarks. The intervention changes output probabilities noticeably, but rarely changes what the model actually predicts. Conclusion: the localization reflects where the model is sensitive to a perturbation, not where the information is fully stored. <p class="citation">Niu et al., 2024</p>
## ROME fails symmetry and synonymy Niu et al. introduce two new evaluation criteria: Symmetry: if you edit "the capital of Canada is Rome," the model should also accept "Rome is the capital of Canada." Synonymy: if you edit a fact about "toothache," the model should generalize to "tooth pain." ROME fails both. The edit is local to the specific surface form used during editing and does not propagate to logically equivalent expressions. This suggests ROME alters token association patterns rather than a stored fact. <p class="citation">Niu et al., 2024</p>
## Methods - Causal tracing (Meng et al.): three-run intervention (clean / corrupted / restore); measure indirect effect per (layer, token) position - Model editing (Meng et al.): rank-one weight update at target layer - Evaluation criteria (Niu et al.): reliability, generality, locality, portability, symmetry, synonymy
## Today's agenda * <span style="color: #767676;">Paper overviews</span> * <span style="color: #767676;">Deep dives</span> * Discussion
## Discussion 1: What is knowledge in a language model? Niu et al. argue that MLPs store "token expression patterns," not knowledge. But what would it mean to store knowledge? Facts, syntax, and co-occurrence statistics all localize similarly. Does the distinction between knowledge and pattern even make sense for a statistical model? What criteria would a representation need to satisfy to count as storing knowledge rather than pattern-matching?
## Discussion 2: What does a successful edit require? ROME achieves good generalization on paraphrases but fails on symmetry and synonymy. What would a truly successful edit look like? Should changing "the capital of Canada" also update all downstream beliefs that depend on that fact? Is model editing fundamentally limited because facts are not stored in one place, or is it a matter of finding a better intervention?
## Discussion 3: Causal tracing and its limits Causal tracing identifies states whose restoration recovers the correct prediction. But Niu et al.'s parallel finding complicates the picture: KN localization (via integrated gradients) identifies neurons with high gradient sensitivity — yet suppressing those neurons rarely overturns the prediction. Both methods identify where a model is sensitive to an intervention, but neither guarantees that the intervention is sufficient to change behavior. What is the relationship between sensitivity and storage? Can a state be causally important without being the site where information is stored?
# Questions?