This work was done during one weekend by research workshop participants and does not represent the work of Apart Research.
AI Testing
Accepted at the 
AI Testing
 research sprint on 
December 19, 2022

Counting Letters, Chaining Premises & Solving Equations: Exploring Inverse Scaling Problems with GPT-3

Language models generally show increased performance in a variety of tasks as their size increases. But there are a class of problems for which increase in model size results in worse performance. These are known as inverse scaling problems. In this work, we examine how GPT-3 performs on tasks that involve the use of multiple, interconnected premises and those that require the counting of letters within given strings of text as well as solving simple multi-operator mathematical equations.

D. Chipping, J. Harding, H. Mannering, P. Selvaraj
4th place
3rd place
2nd place
1st place
 by peer review