# CMSC 25750/35750: ## Large Language Models ## Lecture 12: LLMs as Research Agents <p style="text-align: center;"> Chenhao Tan<br/> University of Chicago<br/> @ChenhaoTan, @chenhaotan.bsky.social<br/> chenhao@uchicago.edu </p>

## Logistics - Project updates

## Models vs. Harnesses A model is the underlying LLM. A harness is everything else: prompting strategy, scaffolding, tool setup, search loop, agent orchestration. Today's papers are almost entirely about harnesses: - AlphaEvolve's evolutionary loop, population pool, self-reflection - Agent Laboratory's three-stage pipeline, mle-solver, co-pilot mode The same base model (e.g., Gemini) can produce very different results depending on the harness around it.

## Why the distinction matters for evaluation Model leaderboards rank models on fixed benchmarks with fixed harnesses. But in practice, harness choice is a free variable. - A weaker model under a better harness can outperform a stronger model under a worse one - Benchmark scores conflate model capability with harness design - Improving the harness is often faster and cheaper than training a better model Thought experiment: how far can we go if we no longer have new models, but can only improve the harness?

## Try ProgramBench Think about the stages of benchmarking from Ofir

## Today's agenda * Paper overviews * Deep dives * Discussion

## Today's agenda * Paper overviews * <span style="color: #767676;">Deep dives</span> * <span style="color: #767676;">Discussion</span>

## Research as the frontier of agents beyond coding Coding agents operate in a closed loop: write code, run it, observe the output, revise. Research extends this into open-ended territory: formulating hypotheses, reviewing literature, designing experiments, interpreting results. Research is harder to evaluate, harder to verify, and harder to bound---which makes it a useful stress test for what agents can and cannot do. Two papers, two angles: AlphaEvolve as an evolutionary coding agent for algorithmic and scientific discovery, Agent Laboratory as a human-in-the-loop research co-pilot.

## AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery Novikov et al. (2025): an evolutionary coding agent where LLMs iteratively edit code, guided by automatic evaluators, to discover novel algorithms and improve mathematical bounds. - Improved 4x4 complex matrix multiplication (98 multiplications, surpassing prior state of the art) - Tightened bounds on over a dozen open mathematical problems (Sidon sets, packing, kissing numbers) - Optimized production infrastructure at Google: data center scheduling, hardware circuit design, LLM training speed - The system improved the LLM that powers itself <p class="citation"><a href="https://arxiv.org/abs/2506.13131">Novikov et al., 2025</a></p>

## Agent Laboratory: Using LLM Agents as Research Assistants Schmidgall et al. (2025): a three-stage pipeline (literature review, experimentation, report writing) that takes a research idea and produces a code repository and paper draft. - o1-preview backend produces the highest-rated reports (3.4/5 utility); o1-mini produces the highest experimental quality (3.2/5) - Human co-pilot involvement raises external evaluation scores from 3.8/10 to 4.38/10 - LLM peer reviewers systematically overestimate paper quality (6.1/10 vs. 3.8/10 human) <p class="citation"><a href="https://arxiv.org/abs/2501.04227">Schmidgall et al., 2025</a></p>

## Today's agenda * <span style="color: #767676;">Paper overviews</span> * Deep dives * <span style="color: #767676;">Discussion</span>

## AlphaEvolve: the core idea Standard LLM code generation produces one candidate. AlphaEvolve maintains a population of programs and uses LLMs to iteratively improve them, guided by automatic evaluators. The evolutionary loop: 1. Sample a high-performing program from the population pool 2. LLM applies EDIT (replace lines n:m) or REPLACE (rewrite file) 3. Compile and evaluate with a scoring function 4. Update the pool if the score improves Errors trigger up to three automatic repair attempts before the candidate is discarded. <p class="citation">Novikov et al., 2025</p>

## What makes it evolutionary Population diversity: sampling from the top-k programs (not just the best) maintains multiple lineages and avoids premature convergence. Self-reflection: the LLM sees an error history (last 5 failures) and a code history (last 2 candidates) before proposing changes. Parallelism: multiple edits are generated in batch each step; the best replaces the lowest-scoring pool member. The evaluator is the selection pressure---any computable objective (accuracy, runtime, mathematical bound) can drive the search. <p class="citation">Novikov et al., 2025</p>

## Matrix multiplication: a concrete result Multiplying two \\(n \times n\\) matrices naively requires \\(n^3\\) scalar multiplications. Strassen (1969) showed two \\(2 \times 2\\) matrices can be multiplied in 7 multiplications instead of 8. For \\(4 \times 4\\) complex-valued matrices, the best known procedure required more scalar multiplications. AlphaEvolve found one using 98---a provably correct improvement over the prior best known result. The algorithm was verified mathematically, not just empirically. <p class="citation">Novikov et al., 2025</p>

## Open mathematical problems AlphaEvolve tightened bounds on over a dozen problems that had been open for years or decades: - Sidon set autocorrelation inequalities: improved upper bound on \\(C_1\\) from 1.5098 to 1.5053 - Kissing number in dimension 11: improved lower bound from 592 to 593 (593 points with integral coordinates satisfying packing conditions) - Heilbronn triangle problem (n=11): improved minimum triangle area from 0.036 to >0.0365 - Packing unit hexagons in a hexagon: improved for n=11 and n=12 Each result is small in isolation. What if they keep running? <p class="citation">Novikov et al., 2025</p>

## Impact on production infrastructure Data center scheduling: AlphaEvolve discovered a more efficient scheduling algorithm for Google's production compute workloads. Hardware accelerator design: found a functionally equivalent simplification in a circuit used in hardware accelerators, reducing complexity. LLM training: accelerated the training pipeline of the LLM that powers AlphaEvolve itself. The last result is notable: an AI system improved the substrate it runs on. This is the buzz word now, but was not heavily celebrated. <p class="citation">Novikov et al., 2025</p>

## Agent Laboratory: the three-stage pipeline Input: a human-provided research idea. 1. Literature review: retrieve and read papers from arXiv 2. Experimentation: design a plan, write and debug code, train models 3. Report writing: produce a LaTeX draft, simulate peer review, refine <p class="citation">Schmidgall et al., 2025</p>

## mle-solver: iterative code refinement The core of the experimentation stage is mle-solver, which operates in a loop: 1. Sample a high-performing program from a pool 2. Generate a new candidate via EDIT (targeted line replacement) or REPLACE (full rewrite) 3. Score the candidate with an LLM reward model (0 to 1, measuring alignment with the research plan) 4. Update the pool if the new candidate scores higher A self-reflection mechanism records error history and past code to guide future iterations. The pool maintains diversity while keeping quality high. <p class="citation">Schmidgall et al., 2025</p>

## Two operating modes Autonomous mode: the system runs end-to-end without human input after the initial idea is submitted. Co-pilot mode: after each stage, the system pauses for human review. The researcher can provide feedback and trigger iteration before the next stage begins. The co-pilot framing is intentional: Agent Laboratory targets tedious execution tasks (coding, writing, data preparation) while preserving human ownership of ideation and critical design choices. <p class="citation">Schmidgall et al., 2025</p>

## What the numbers say Human evaluation (10 PhD student reviewers, 5 research topics, 3 backends): | Backend | Exp. quality | Report quality | Usefulness | |---|---|---|---| | GPT-4o | 2.6 / 5 | 3.0 / 5 | 4.0 / 5 | | o1-mini | 3.2 / 5 | 3.2 / 5 | 4.3 / 5 | | o1-preview | 2.9 / 5 | 3.4 / 5 | 4.4 / 5 | NeurIPS-style scores (human reviewers): 3.5--4.0 / 10. Papers accepted at NeurIPS average 5.85 / 10. Co-pilot external evaluation: 4.38 / 10 vs. 3.8 / 10 autonomous. <p class="citation">Schmidgall et al., 2025</p>

## The self-evaluation gap Automated peer review (LLM-based): average 6.1 / 10. Human peer review: average 3.8 / 10. A gap of 2.3 points. LLMs reviewing LLM-written papers systematically overestimate quality. This matters because the report writing pipeline uses LLM-simulated peer review to decide whether to refine or finalize. If the feedback signal is inflated, the system may declare a paper done well before a human would. <p class="citation">Schmidgall et al., 2025</p>

## Failure modes worth knowing Hallucination: some GPT-4o papers reported experimental results and hyperparameters for runs that were never executed. Instruction-following: the literature review agent repeatedly calls `summarize` instead of moving on, consuming steps until the limit is reached. Silent failure: mle-solver sometimes produces code with 0% accuracy and does not self-correct. Process termination: the system occasionally generates Python `exit()` calls, ending the entire run. <p class="citation">Schmidgall et al., 2025</p>

## Today's agenda * <span style="color: #767676;">Paper overviews</span> * <span style="color: #767676;">Deep dives</span> * Discussion

## Discussion 1: AI self-improvement Jack Clark (Anthropic co-founder) recently claimed that AI self-improvement is coming soon: models helping design better models, agents improving their own training pipelines. AlphaEvolve already did a version of this---it accelerated the training of the LLM that powers it. What would meaningful AI self-improvement look like? What are the limits?

## Discussion 1: tentative answers - AlphaEvolve's self-improvement is narrow: the objective (training speed) was fixed by humans, and the system optimized within it---this is closer to autotuning than open-ended improvement - With a perfectly honest metric, typical skeptical arguments: diversity collapse (a self-training system narrows its own distribution over iterations) and resource limits (compute and data are finite) - But both concerns may be weaker than assumed: synthetic data and tool use can sustain diversity; compute has scaled further than most predictions; AlphaEvolve shows that automated evaluators can provide signal without human-generated data

## Discussion 1: tentative answers (cont.) - Meaningful self-improvement would require the system to improve its own evaluators or redefine its objectives, not just optimize a fixed metric faster - A key limit is the evaluation bottleneck: you can automate code execution, but you cannot easily automate "is this a better research direction?" - Compounding error: if the scoring function has blind spots, iterated self-improvement can drift far from what we actually want, with no external signal to catch it - A harder concern: even if capability improves, it is not clear that judgment, reliability, or alignment improve alongside it---a more capable system optimizing the wrong thing is more dangerous, not less

## Discussion 2: What should open-ended verification look like? AlphaEvolve's results are verifiable: a matrix multiplication is correct or not, a bound is tight or not, an infrastructure metric goes up or down. Agent Laboratory's results are not: paper quality is judged by humans, and LLM judges inflate scores by 2.3 points. What would a verification framework for open-ended research outputs look like?

## Discussion 2: tentative answers - For mathematics: formal proof assistants can certify results mechanically without understanding the discovery---this is the strongest form of verification - For empirical ML: independent replication on held-out data is necessary; hallucinated results in Agent Laboratory papers would fail immediately, but replication is costly and rarely done - For open-ended claims (paper quality, research significance): there is no automatic verifier, and LLM judges are unreliable---human review remains the only credible signal - The general principle: the verifiability of the output domain should set the bar for how much autonomy we grant the system

# Questions?