Neuron to Graph: Interpreting Language Model Neurons at Scale

N2G distills neuron behavior into interpretable graphs using truncation and saliency methods. Graphs aid manual interpretation and automate validation. N2G scales interpretability in LLMs, facilitating research automation.

Understanding the function of individual neurons within language models is essential for mechanistic interpretability research. We propose Neuron to Graph (N2G), a tool which takes a neuron and its dataset examples, and automatically distills the neuron's behaviour on those examples to an interpretable graph. This presents a less labour intensive approach to interpreting neurons than current manual methods, that will better scale these methods to large language models (LLMs). We use truncation and saliency methods to only present the important tokens, and augment the dataset examples with more diverse samples to better capture the extent of neuron behaviour. These graphs can be visualised to aid manual interpretation by researchers, but can also output token activations on text to compare to the neuron's ground truth activations for automatic validation. N2G represents a step towards scalable interpretability methods by allowing us to convert neurons in an LLM to interpretable representations of measurable quality.

An interview with

Interpreting language model neurons at scale
" was written by

Author contribution

No items found.


Send feedback

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Media kit


No items found.

All figures

No items found.