
MedChemBench: Benchmarking AI for Medicinal Chemistry Reasoning

MedChemBench is a benchmark for evaluating AI models on reasoning tasks drawn from medicinal chemistry research. It tests whether models can connect a molecule’s structure with experimental evidence to explain results, compare compounds, and predict properties, with the aim of helping scientists develop better medicines. The benchmark is motivated by the tradeoff problem at the center of drug discovery: improving one property of a molecule can worsen another, and medicinal chemists must design, make, and test related compounds to decide what to try next.
MedChemBench contains 99 problems drawn from 117 papers and patents and covers 761 distinct molecular structures. For most problems, models receive the context available at a chosen point in a research program: the biological target, the researchers’ goals, the compounds studied so far, and selected experimental findings. Four components are graded: reconstructing molecular structures with correct atom connections and stereochemistry, providing scientific reasoning supported by the supplied evidence, ranking and selecting compounds that meet stated requirements, and predicting property values that agree with reported measurements and categories. Predictions earn partial credit based on closeness to measured values, and measurements already supplied in the prompt must be reproduced correctly.
One problem illustrates how competing properties are tested. Using research on inhibitors of PRMT5, a protein studied in cancer drug discovery, the model receives structures and earlier experimental results and must predict how three related compounds act in cells, cross a cell barrier, and behave in a liver-enzyme assay. Withheld later experiments showed a tradeoff: compound 23 blocked PRMT5 in cells at a lower concentration, while compound 24 crossed the cell barrier more readily and broke down more slowly in the liver-enzyme assay. The task evaluates whether models can connect molecular structures to several aspects of compound behavior at once, a central challenge because stronger target activity does not necessarily bring better permeability or slower breakdown.
MedChemBench also tests spatial reasoning during structure reconstruction. In one problem, models must assemble a molecule from a central structure and two attached groups while preserving an 18-atom ring. At the xhigh reasoning setting shared by all eleven models, GPT-6 Astra reconstructed the compound correctly in all ten responses; GPT-5.6 Sol produced a 15-atom ring, and GPT-5.6 Luna split the molecule into two disconnected fragments. Selected answers for compound 17 show a correct reconstruction, a smaller ring, and disconnected fragments, demonstrating that these errors change the identity of the molecule being analyzed. For the structure component overall, Astra earned 95.4% of available credit at xhigh, showing strong performance at reconstructing the supplied molecules.
At the xhigh reasoning setting, GPT-6.2 Astra earned 84.5% of available credit, compared with 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol, and 58.2% for GPT-5.6 Sol. Across the full benchmark, GPT-6 Astra earned 83.3% at the shared xhigh setting and reached 83.8% at max effort; GPT-6.1 Terra earned 75.1% in the same comparison. Higher reasoning settings improved scores, while large differences between models remained. A plot of performance versus token usage across 51 combinations of nine models from GPT-5.6 onward shows this pattern. Component results separate four abilities: GPT-6 Astra was strongest at reconstructing molecules and explaining scientific evidence, but it earned only about half of available credit for selecting compounds and predicting numerical properties. The results do not establish whether models can discover new drugs or improve laboratory decisions. Prior exposure to the published sources cannot be ruled out, and each response was a single attempt without tools, feedback, or new experiments. MedChemBench evaluates reasoning from historical evidence rather than real-world drug discovery, so its value is in identifying where models are advancing and where challenges remain.

