This work was done during one weekend by research workshop participants and does not represent the work of Apart Research.
Accepted at the 
 research sprint on 
July 16, 2023

Factual recall rarely happens in attention layer

In this work, I investigated whether factual information is saved only in the FF layer or also in the attention layers, and found that from a large enough FF hidden dimension, factual information is rarely saved in the attention layers.

Bary Levy
4th place
3rd place
2nd place
1st place
 by peer review