Every Eval Ever and Hugging Face Community Evals Now Cross-Compatible

AI evaluation results are a mess. The same model on the same benchmark yields wildly different scores depending on who ran it and how — LLaMA 65B has been reported at both 63.7 and 48.8 on MMLU — because evaluation settings like generation config, harness version, and reproducibility notes are commonly unreported. Results are scattered across papers, leaderboards, blog posts, and harness logs, each in its own format, making it nearly impossible for users, researchers, and policymakers to trust, understand, and compare models. Every Eval Ever (EEE) launched in February 2026 as a cross-institutional effort to fix the reporting side, while Hugging Face Community Evals launched simultaneously to decentralize how benchmark scores get displayed on the Hub.

EEE provides one JSON schema that records who ran the evaluation, which model, how it was accessed, generation settings, what the metric actually means, and optionally a companion JSONL file for per-sample outputs. The datastore on Hugging Face has grown to roughly 229,000 evaluation results across more than 22,000 models and 2,200 benchmarks, pulled from 31 different reporting formats. Reproducing those runs from scratch would cost hundreds of thousands of dollars, making a strong case for not letting the data scatter once someone has paid to generate it. The new converter maps EEE records into the small YAML files Hugging Face expects, so submitters don’t need to maintain the same result in two formats by hand. When you submit through your organization’s official Hugging Face account, results show up with a verified checkmark on EvalEval, signaling the numbers come straight from the source.

For builders and researchers, the takeaway is practical: you can now send evaluation results to both Community Evals and EEE with a single workflow, and the same evaluation ends up visible on model pages and fully interpretable via structured records. The converter handles field mapping, hash-checking, and conflict detection, then writes local YAML previews and only opens PRs after your explicit sign-off. This cross-compatibility patches the gap between where people look at models (Hugging Face) and where the full, reproducible record lives (EEE). If you’re running evaluations, contributing your results to both destinations makes your numbers legible and trustworthy rather than just another scattered score.