Transformers enabled PyTorch’s scaled dot-product attention (SDPA) for T5 on July 30, 2026. As a side effect, UMT5‘s decoder could attend to future tokens, letting it peek at the answer during training. Nothing crashed, and the bug stayed in main for almost two months. Replaying the commits with a small UMT5 model on CPU flagged it on every one. The post describes that model and the Model Integration Contract tests built around it.

The full test suite is too heavy for every PR: Transformers supports over 750 model types, with 88,000+ CPU tests on main and 200,000+ GPU tests in a daily run. Integration tests load real checkpoints and run on real hardware, so running them on every PR isn’t practical. Instead, they shrink the model: a fixture from the real checkpoint’s configuration, with fewer and narrower layers but every kind of layer it has, and unchanged tokenizer and processor files. It is published once and reused in fast integration tests on CPU. The UMT5 fixture used 66 MB of weights; the source checkpoint is 1,227 MB. This also covers very large models that don’t fit on standard GPU runners. The goal is checks beyond loading and a forward pass; the UMT5 bug would have passed a forward pass.

A Model Integration Contract (MIC) has three parts: a fixture, a usage path (a runnable Markdown example like generating text or a training step), and baselines (recorded outputs, reviewed by a human). The contract holds if running the usage path on the fixture still produces the baselines. Contracts cover an architecture and usage path, not a single checkpoint; compatible checkpoints can share a fixture while each keeps its own expected results. They are published in huggingface-mic. The SmolLM contract generates from prompts and compares token IDs with a recorded answer, and also checks training loss, finite gradients, and that an optimizer step changes weights. A fleet dataset on the Hub pins each MIC to a frozen revision and recording package versions; every CI run records the fleet revision used, so results are reproducible. On a PR, MICs are chosen based on the files the patch changes.

Replaying frozen MICs over five months of commits, the small UMT5 fixture flagged the regression from its introduction on July 30, 2026 until its fix on September 28. Label log-probabilities differed from the baseline by 1.22, and training loss by about 0.20. One detail mattered: the contract used SDPA, the default attention path; the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint. Reverting the fix on a proof branch failed with the same differences, and the selected T5 contract’s runner step took 13.4 seconds on CPU, excluding setup.

The shrinking recipe has four steps. Shrink the configuration: reduce hidden sizes, attention heads, and depth while keeping the different layer kinds, including mixtures of sliding/full attention or dense/MoE layers. Some architectures need extra rules; the ModelTester in test_modeling_.py supplies sizes for widths the generic rule doesn’t know. Keep preprocessing unchanged: preserve vocabulary, special tokens, and input dimensions, and copy tokenizer/processor files byte for byte. Build and validate weights: try seeded random weights and change inputs to check the model responds; if not, train it on the contract’s small batch. Reload and check outputs, loss, and gradients are finite. Freeze and record: save accepted weights, measure repeatable outputs on a known-good build, review baselines per platform and precision, and publish a pinned snapshot. Keep the original weights file format so the conversion code users hit when loading the real checkpoint is exercised. The Llama 4 Scout shrink report shows 108.6B parameters becoming 44.7M: vision 34 layers to 2, language 48 to 4.

The CLI tmic creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure candidate baselines. The LLM drafts but doesn’t approve; changed expectations require human review. tmic run org/model –local-only –fixtures-only –no-history –bill-to ” builds and records baselines locally, asking review at each step and stopping before upload. Drafts come through Inference Providers, billed to the user’s account. Agents built MICs across model families; edge cases were fixed in the recipe itself, with help from model experts.

With random weights, the Whisper fixture barely responded to audio changes, so it was overfit on a few tones paired with words like “red” and “green”—not speech recognition, just making audio changes visible. In the weight-initialization study, random Whisper fixtures caught 18 of 24 injected faults; overfit fixtures caught all 24. That doesn’t settle which is better: those were synthetic faults, and no real regression has been found that only overfit catches. Both versions missed an assisted-decoding bug exposed by the real Whisper checkpoint. So random weights are used when they pass checks, overfit when better sensitivity or readable answers are needed; training takes seconds but is one more step that can fail to converge.

As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs. Whether it should block depends on unmeasured questions: real bugs caught on PRs vs. replays, false alarm rate, and baseline review effort. It’s also unknown how many bugs only show with real weights, and whether overfitting catches real bugs random weights miss. Coverage is partial across model families and APIs. When a regression slips through, the missing usage path can be added and replayed on broken and fixed versions. Future work includes linking original Hub model pages to their MICs, so developers or agents using a checkpoint can start from an example tested on that checkpoint, its environment, and its expected outputs.