# CMSC 25750/35750: ## Large Language Models ## Lecture 14: AI and Human Complementarity <p style="text-align: center;"> Chenhao Tan<br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p>

## Logistics - Project updates

## Today's agenda * Paper overviews * Deep dives * Discussion * Course review

## Today's agenda * Paper overviews * <span style="color: #767676;">Deep dives</span> * <span style="color: #767676;">Discussion</span> * <span style="color: #767676;">Course review</span>

## Two questions about AI and human capability Does exposure to AI that performs beyond human level push humans to perform better? Does using AI to get work done cost you the ability to do that work yourself? Today's papers give evidence for both. One is a natural experiment over 71 years; the other is a controlled experiment over one hour. Both center on the same mechanism: what happens to human capability when a powerful tool does part of the work.

## Superhuman AI Can Improve Human Decision-Making by Increasing Novelty Shin et al. (2023): 5.8 million professional Go moves from 1950-2021. AlphaGo's 2016 victory provides a clean before/after comparison. - Human decision quality was flat for 66 years, then rose sharply after 2016 - The improvement is driven by novel moves---sequences never seen before in the historical record - Novel moves contributed more to post-AI quality gains than non-novel moves (interaction \\(\beta_3 = 0.515\\), p < 0.001) - The improvement is not memorization: it appears in late-game positions and in responses to off-script opponent moves <p class="citation"><a href="https://www.pnas.org/doi/10.1073/pnas.2214840120">Shin et al., 2023</a></p>

## How AI Impacts Skill Formation Shen and Tamkin (2026): a randomized experiment with 52 professional developers learning a new Python library. Half used web search only; half also had access to GPT-4o. - AI users scored 17% lower on a knowledge quiz (4.15 points on a 27-point scale; Cohen's d = 0.738, p = 0.010) - No statistically significant speedup from AI assistance - Six interaction patterns identified: three correlated with high quiz scores, three with low - The learning gap comes from reduced error encounters, not reduced time on task <p class="citation"><a href="https://arxiv.org/abs/2601.20245">Shen and Tamkin, 2026</a></p>

## Today's agenda * <span style="color: #767676;">Paper overviews</span> * Deep dives * <span style="color: #767676;">Discussion</span> * <span style="color: #767676;">Course review</span>

## The natural experiment On March 15, 2016, AlphaGo defeated world champion Lee Sedol 4-1. This is a precise, documented date at which superhuman AI became publicly known and accessible to study. Shin et al. use this as an exogenous shock: everything before is pre-AI exposure, everything after is post. Data: 5.8 million moves from professional games in the Games of Go on Disk (GoGoD) database, 1950-2021. Move quality is evaluated by KataGo, a superhuman AI that simulates 10,000 game continuations per move. <p class="citation">Shin et al., 2023</p>

## Measuring decision quality and novelty Decision Quality Index (DQI): for each human move, KataGo computes the win-rate difference between the human's move and the optimal move. DQI = 100 minus that gap. Higher is better. Novelty Index: the move number at which a game first creates a sequence never seen before in the historical dataset. Earlier novelty (fewer moves into the game before departing from known patterns) corresponds to a higher novelty index. Both indices are constructed at the move level, across 71 years of professional play. <p class="citation">Shin et al., 2023</p>

## 66 years flat, then a step change <img src="images/go.png" style="max-width: 80%;">

## 66 years flat, then a step change Human professional play showed almost no improvement from 1950 to 2016---66 years of stable performance on DQI. After 2016, quality rose sharply. The novelty index also increased: games departed from historically known sequences earlier, and this shift coincides with the AlphaGo shock. The pattern holds after controlling for player fixed effects, move number, and time period. The implication: improvement in this domain was not gradual or inevitable. It required an external catalyst. <p class="citation">Shin et al., 2023</p>

## Novelty as the mechanism The key finding is an interaction: novel moves contributed more to the post-2016 quality improvement than non-novel moves. Interaction coefficient: \\(\beta_3 = 0.515\\) (p < 0.001). A novel move post-2016 is associated with a larger quality gain than a non-novel move in the same period. More than 50% of novel moves appeared before move 10. By move 21, 99% of opening sequences had become historically novel. Players were not playing more conservatively after AlphaGo. They were exploring earlier and more systematically. <p class="citation">Shin et al., 2023</p>

## Ruling out memorization The intuitive alternative explanation: professionals simply memorized AlphaGo's moves and replayed them. Evidence against: - Quality improved even on moves that differed from KataGo's recommendations---humans got better at positions where AI would play differently - Quality improved in late-game positions (moves 21-60) where memorization is not feasible - Quality improved in responses to opponent deviations from optimal AI play---off-script situations - Backdating 600,000 simulated AI decisions to pre-2016 did not produce the same novelty improvement The mechanism appears to be conceptual: AI broke humans' assumptions about which moves were worth exploring. <p class="citation">Shin et al., 2023</p>

## Shen and Tamkin: experiment design 52 professional and freelance developers. Task: learn and use Python Trio, an asynchronous programming library. Chosen because it is obscure enough that participants are unlikely to have prior knowledge. Control condition: web search and documentation only. Treatment condition: same resources plus GPT-4o in a chat interface. Both groups: 35 minutes total, two tasks. A warm-up task with no AI access calibrates Python familiarity. A 14-question quiz (27 points) covering conceptual understanding, code reading, debugging, and code writing follows. All 51 sessions were recorded for qualitative analysis. <p class="citation">Shen and Tamkin, 2026</p>

## Six interaction patterns From screen recording analysis, three patterns predicted low quiz scores (24-39%): - AI delegation: copy-paste AI output without reading it - Progressive reliance: start with documentation, switch entirely to AI after the first error - Iterative debugging: send error messages to AI in a loop, applying fixes without understanding them Three patterns predicted high quiz scores (65-86%): - Generation-then-comprehension: request AI code, then manually type it to understand each line - Hybrid: alternate between writing code and asking for explanations - Conceptual inquiry: ask AI to explain concepts, write code independently <p class="citation">Shen and Tamkin, 2026</p>

## The results Quiz scores: AI group 17% lower (4.15 point gap on 27-point scale; Cohen's d = 0.738, p = 0.010). Task completion time: no significant difference. AI group mean ~22 min; control group mean ~23 min. Error encounters: control group encountered a median of 3 errors per task; AI group encountered 1. The control group hit significantly more Trio-specific errors requiring conceptual understanding. AI query distribution (25 treatment participants): 79 explanation queries, 51 generation queries, 9 debugging queries. Only 16 of 25 participants requested generated code at all. <p class="citation">Shen and Tamkin, 2026</p>

## What the quiz gap reveals The largest gap appeared on debugging questions---the skill most dependent on having worked through errors yourself. The AI did not save time: heterogeneous usage patterns (some participants spent over 30% of task time composing queries) offset the speed of AI-generated code. The mechanism is not that AI users spent less time on task. It is that they spent that time differently: interacting with AI rather than wrestling with compiler feedback and documentation. Struggle with errors appears to be pedagogically load-bearing, not just friction. <p class="citation">Shen and Tamkin, 2026</p>

## An important caveat The paper measures skill after one hour of learning. It does not follow participants over weeks or months. The authors note this is likely a lower bound: agentic tools that require even less explicit human involvement would amplify the learning deficit. But the direction of the effect could also reverse over longer timescales---if AI tools free up time that developers use for harder, more conceptually demanding work. What the experiment establishes: in the short run, AI assistance during learning trades skill acquisition for the appearance of productivity. <p class="citation">Shen and Tamkin, 2026</p>

## Today's agenda * <span style="color: #767676;">Paper overviews</span> * <span style="color: #767676;">Deep dives</span> * Discussion * <span style="color: #767676;">Course review</span>

## Discussion 1: Are the two papers in conflict? Skill formation: AI assistance during learning reduces knowledge acquisition. Go novelty: exposure to superhuman AI improves human decision quality. Both can be true. But they point in opposite directions on whether AI benefits human capability. What is different about the two settings that explains the divergent outcomes?

## Discussion 1: tentative answers - Mode of engagement: developers used AI as a real-time assistant during task completion; Go professionals studied AI games as a training input, then performed independently in competition - Stakes and feedback: professional Go has an unambiguous win/loss signal that immediately reveals whether a novel move was good; a software developer can ship buggy code without knowing - Timescale: Go improvement emerged over years of deliberate practice post-AI; the skill formation study measured a 35-minute session - The synthesis: AI used as a substitute for thinking impairs learning; AI used as a source of examples to study and internalize may accelerate it---the difference is whether the human does the cognitive work

## Discussion 2: How do we build AI that complements human capability? The Go study shows AI can expand what humans think is possible. The skill formation study shows AI can quietly erode the ability to do things yourself. Both results come from the same technology. The difference is in the design and context of use. What would a roadmap for AI that reliably complements rather than substitutes for human capability look like?

## Discussion 2: tentative roadmap * Measure longitudinal skill outcomes, not short-run productivity. * Design AI that tracks cognitive engagement: detect delegation patterns, require explanation before generation, introduce friction at the moments learning would otherwise happen. * Build AI that understands context and task difficulty: distinguish learning from routine execution, and adapt the mode of assistance accordingly. * Develop AI that closes the feedback loop faster and anticipates failure modes.

## Today's agenda * <span style="color: #767676;">Paper overviews</span> * <span style="color: #767676;">Deep dives</span> * <span style="color: #767676;">Discussion</span> * Course review

## Unit 1: Interpretability Takeaway: MechInterp is cool and CoT can be unreliable. Open problem: how do we scale mechanistic methods from toy circuits to real behaviors without losing precision?

## Unit 2: Alignment Takeaway: You are not really worried about alignment. Open problem: what are the conditions or observations that indicate that we are too late to worry about alignment?

## Unit 3: Agents Takeaway: AI agents are promising but are complex systems that are different from humans and cause side effects. Open problem: Building benchmarks and evaluation for open-ended problems, e.g., scientific discovery.

# Questions?