This paper announces the results of a large study to evaluate the ability of AI agents to solve unpublished research-level problems in theoretical physics. If you are at all interested in physics, please take a look.
The abstract should be self explanatory. I’ve been thinking about these models quite closely for a while. But, frankly, I wasn’t expecting this day to come as soon as it has. After we found these results, I couldn’t sleep properly for a couple of nights.
Let me add some back story since the timeline is revealing.
About 6 months ago, just for fun, I prompted a frontier agent with an unpublished problem whose answer I had worked out. The agent got the answer wrong but it was impressive that it managed to make any progress at all, and could solve the problem after a few hints.
Praneeth suggested it might be fun to collect a suite of such problems along the lines of the mathematicians’ First Proof assessment. Soon we had a small core group and started requesting experts across the world to contribute problems. Problem collection was a slow process, since we were making a significant ask of contributors. We wanted unpublished problems, with clean verifiable answers. So, our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us.
Even as problems started coming in, the agents started improving as newer models were released. We kept urging people to push up the difficulty of their problems. But, there is a ceiling to this difficulty since even world experts have only so-many problems lying about whose answers they know and haven’t published.
I don’t know of any individual who can solve any significant fraction of our final collection of problems without AI assistance. For instance, I can solve the problems I contributed myself and understand a few others. Most problems don’t even make sense to me.
Our initial evaluation suggested that GPT-6-Astra’s score was in the late eighties followed closely by Opus 5.5
But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra’s score started to inch up and it finally ended up with 98% (49/50).
The one problem it doesn’t solve is not because it makes a “mistake”. This is a trick-prompt, and requires the model to make an inspired guess, and then bend the rules a little. It can’t do that yet.
We comment a little on the broader implications in our conclusions. Clearly, these agent are enormously powerful tools that will accelerate discovery in the short run. They also raise serious structural concerns about access, equity and pedagogy.