Core dump epidemiology: fixing an 18-year-old bug
OpenAI‘s Rockset service, a key part of ChatGPT’s data infrastructure, experienced mysterious crashes where C++ functions returned to bogus addresses. The initial debugging approach—inspecting individual core dumps in “doctor mode”—failed because the team mistakenly assumed all crashes had a single cause. The crashes appeared to violate normal failure modes: return addresses were NULL or the stack pointer was misaligned by 8 bytes. This made the bug seem impossible, as plausible explanations like stray writes, signal delivery, or kernel bugs were ruled out one by one.
The turning point came when the team switched to an “epidemiologist” approach: building an automated pipeline to analyze all core dumps from the past year. This immediately revealed two separate crash clusters. One cluster—misaligned-stack crashes—traced back to a single physical Azure host with silent hardware corruption. The other cluster—return-to-null crashes—occurred during C++ exception unwinding. Further investigation uncovered an 18-year-old race condition in GNU libunwind‘s x86_64 setcontext implementation: a one-instruction window where a signal could overwrite the synthesized return address after the stack pointer was updated. The bug had always existed but only became visible when Rockset‘s combination of high exception rates, frequent timer signals, and a recent increase in handler stack usage crossed a threshold.
The takeaway for serious builders is that building high-quality population data is often more valuable than deep expertise in obscure internals. Once the team had accurate labels for each crash, the structure of the problem became obvious. This reinforced a commitment to deep instrumentation and automated investigation. Reliability isn’t just about fixing bugs—it’s about building the data, workflows, and skills that turn impossible problems into diagnosable and solvable ones. The immediate fix was switching to libgcc’s unwinder and upstreaming a reproducer to GNU libunwind.
