SemiAnalysis materials-science test: Astra beat a published reference on one subtask but scored 28%

Interview participants described an evaluation they conducted of models on materials-science tasks. One said Astra solved one subtask better than a published state-of-the-art reference. He characterized model performance as “spiky” across areas and said that in some cases models reach above-superhuman baselines. However, the best Astra materials-science model scored only about 28% on their benchmark.

The speaker said it is difficult to establish a human baseline for a test like this: tasks may require teams of researchers or hardware engineers, and it is hard to estimate a team’s work in hours or days. He specifically noted that the chip-design task would involve many people, not a single engineer. Participants also argued that buyers want models with superhuman capabilities in particular domains, rather than models merely comparable to ordinary people. The conversation did not provide details on the methodology, benchmark composition or independent validation.