Nick Cammarata: Neural networks may be more interpretable than expected

Nick Cammarata of Goodfire argues that modern tools are changing how researchers view neural-network interpretability. He says an early automated interpretability project at OpenAI made only about 2% of neurons legible with the primitive tools then available. Now, he says, the vast majority of neurons in models with up to trillions of parameters are interpretable, and many supposed examples of tangled representations were misunderstood. Cammarata predicts ambitious interpretability will be achieved by the time it is most needed, possibly quite soon as automated researchers improve.

The discussion described mechanistic interpretability as examining what happens inside a model at the level of individual neurons. The earlier superposition hypothesis held that a neuron might respond to many features; sparse autoencoders were used to recover a representation in which each feature has its own neuron. As a participant summarized the new work, that approach may no longer be necessary: researchers can inspect the neurons themselves. This could help not only with safety but also with improving model capabilities, such as mathematics.

The hosts called the work a potentially important scientific moment and suggested interpretability could make alignment more like a conventional science. The discussion did not independently verify Cammarata’s claims; they were presented as his conclusions and the hosts’ assessment. Goodfire was described as an independent mechanistic-interpretability lab. The hosts speculated it might work on open models or serve as an outside auditor for labs, but these were possibilities, not announced arrangements.