In July 2026, Hugging Face and alphaXiv ran a hackathon where over 1,200 community members used coding agents (Claude Code, Codex, Cursor, etc.) to reproduce papers from ICML 2026, claim by claim. Over 19 days, participants published 6,816 reproduction logbooks covering 2,226 papers (34% of the conference).

The challenge was motivated by the scale of modern AI research: ICML 2026 accepted 6,352 papers, roughly double the previous year, while reviewing capacity has not kept up. The authors argue that the same AI agents driving the flood of submissions can also help audit it.

Each reproduction produced a Trackio logbook (a static Hugging Face Space with code, artifacts, and optional agent trace). An automated Logbook Judge (GLM-5.2) issued per-claim verdicts: verified, falsified, toy, or inconclusive. 2,962 cloud jobs were launched on Hugging Face.

Aggregated results: 51% of examined papers (1,103) had at least one claim independently verified. 266 papers were fully reproduced (every claim verified). 632 more were partially reproduced with nothing falsified. 23% (496 papers) had at least one claim falsified or contested, including 49 papers where all claims were falsified and 242 where independent teams reached opposite verdicts on the same claims.

Several notable falsifications were confirmed after adversarial re-verification. The paper “Towards Optimal Robustness in Learning-Augmented Paging” claimed robustness H_k + O(1); a participant found the additive term grows like 0.38 ln k, and re-implementation at larger scales confirmed H_k + Θ(log k) at roughly nine sigma. In “Attention’s forward pass and Frank-Wolfe,” three independent teams found counterexamples to the claimed collapse to origin, with violations first appearing at t = 224, ~3,800, and 6,416 steps. The authors confirmed the same day and are working on a fix. “Self-Distillation Enables Continual Learning” analyzed reverse KL theoretically but released code computing forward KL; the reproduction also failed to reproduce the headline +4pp result. “Do Transformers Need Three Projections?” had ~66% of evaluated label positions as EOS padding tokens with near-zero loss, deflating perplexity roughly threefold; the abstract’s “3.1% quality cost for 50% cache reduction” becomes ~9.4% corrected.

False falsifications also occurred: one logbook claimed a method was “2x slower than baseline” due to a unit mismatch; correctly normalized, the participant’s own data confirmed the claimed 8x speedup.

The authors are contacting authors of confirmed findings. Early responses have been positive; two arXiv corrections are in flight.

On the role of humans: pure agent execution hit limits (local loops, scale-dependent misreads, unit errors). The most reliable results came from human-in-the-loop workflows where participants steered, questioned assumptions, or made judgment calls. One human-in-the-loop winner had the agent build a review UI and the human personally judged 128 image pairs. The authors conclude: the human role is to “manage intelligence effectively,” much like a professor setting up an environment for grad students.

All logbooks, verdicts, traces, and artifacts are public. The authors hope this is the largest open claim-by-claim audit of an ML conference and that it won’t hold the record for long.