The hackathon is happening right now! Join by signing up below and be a part of our community server.
Apart > Sprints

Computational Mechanics Hackathon!

--
No items found.
Signups
--
Entries
June 1, 2024 12:30 AM
 to
June 3, 2024 11:30 PM
 (UTC)

Organized by PIBBSS and Simplex.

Hackathon starts in
--
Days
--
Hours
--
Minutes
--
Seconds
Sign upSign up
This event is finished. It occurred between 
June 1, 2024
 and 
June 3, 2024
✨ Overview
🗺️ Resources
⏰ Schedule
💡 Ideas
🌏 Locations
✉️ Submission
🥇 Entries (
--
)
❓ Emails

Join us for an exciting hackathon where we'll leverage Computational Mechanics to understand and control neural network behavior. This new AI Safety and interpretability approach is built upon a rigorous mathematical framework from physics and enables precise predictions about the internal geometry of AI systems, overcoming limitations of current interpretability methods.

We have $1,500 in prizes for the best projects! $750 for 1st place, $500 for 2nd place, and $250 for third place!

‍
Collaborate with researchers and enthusiasts to stress-test the approach, create benchmarks, and develop a science of AI cognitive capabilities and risks. Contribute to the advancement of AI safety and help build a safer, more controllable AI future.
Sign up now and get ready to contribute to this new approach to AI Interpretability and Safety, and for your chance to win cash prizes!

Sign up here to stay updated for this event

Society is building AI systems with ever-increasing capabilities and interactions with society at large. In order to make sure our future is a healthy and safe one, we need to understand the principles and mechanisms by which these intelligent systems operate. In this hackathon we will use Computational Mechanics, a framework from physics that studies the computational structure of prediction, to do interpretability research on neural networks.

‍

Our initial findings have found that Computational Mechanics can make precise and unintuitive predictions about the internal geometry of activations in transformers. This finding has opened up many avenues of inquiries, and there are many low hanging fruits to pick! Participation in this hackathon requires little background, and can take the form of computational experiments, theory/math, or coding/engineering. In the resources section you can find getting started guides, an open problems document, and a number of coding examples to get you started!

Rules

You will participate in teams of 1-5 people and submit a project on the entry submission page. Each project consists of multiple parts: 1) The PDF report, 2) a maximum 10-minute video overview, 3) title, summary, and descriptions.

‍
You are allowed to think about your project and engage with the starter resources before the hackathon starts but your core research work should happen during the duration of the hackathon.
Besides these two points, this hackathon is mainly an opportunity to meaningfully engage in Computational Mechanics, Interpretability Research, and AI Safety.

Speakers & Collaborators

No items found.

Blog Posts and Documents

Code

Video

  • Jim Crutchfield's lecture series on Computational Mechanics, from his Physics of Information and Physics of Computation courses at UC Davis. Many topics and a great overview of the field. We suggest browsing the list of lectures just to get a sense of the many different parts of this framework - https://csc.ucdavis.edu/~chaos/courses/poci/Lectures/

Relevant Papers

‍

See the updated calendar and subscribe

Time is presented in Pacific Daylight Time (California). 10:30 PDT is 17:30 UTC.

‍
Friday, May 31, 10:30 AM, PDT: Keynote talk with Adam Shai and Paul Riechers to inspire your projects and provide an introduction to the topic. Following the talk we will chat in the discord to discuss teams.

‍
Saturday and Sunday, June 1 and 2, 10:30 AM, PDT: Project discussion sessions on the Discord server.

‍
Sunday, June 2, at 17:30 PDT: Online ending session

‍
Wednesday, June 5, at 10:30 PDT: Project presentations

📍 Registered jam sites

Beside the remote and virtual participation, our amazing organizers also host local hackathon locations where you can meet up in-person and connect with others in your area.
Register the first event below!

🏠 Register a location

The in-person events for the Apart Sprints are run by passionate individuals just like you! We organize the schedule, speakers, and starter templates, and you can focus on engaging your local research, student, and engineering community. Read more about organizing.
Uploading...
fileuploaded.jpg
Upload failed. Max size for files is 10 MB.
Thank you! Your submission has been received! Your event will show up on this page.
Oops! Something went wrong while submitting the form.

📣 Social media images and text snippets

No media added yet
No text snippets added yet

Submit your entry on this page!

Here is the submission template

Uploading...
fileuploaded.jpg
Upload failed. Max size for files is 10 MB.
Uploading...
fileuploaded.jpg
Upload failed. Max size for files is 10 MB.
Uploading...
fileuploaded.jpg
Upload failed. Max size for files is 10 MB.
You have successfully submitted! You should receive an email and your project should appear here. If not, contact operations@apartresearch.com.
Oops! Something went wrong while submitting the form.

See submissions here.

Unsupervised Recovery of Hidden Markov Models from Transformers with Evolutionary Algorithms
Prior work finds that transformer neural networks trained to mimic the output of a Hidden Markov Model (HMM) embed the optimal Bayesian beliefs for the HMM's current state in their residual stream, which can be recovered via linear regression. In this work, we aim to address the problem of extracting information about the underlying HMM using the residual stream, without needing to know the MSP already. To do so, we use the R^2 of the linear regression as a reward signal for evolutionary algorithms, which are deployed to search for the parameters that generated the source HMM. We find that for toy scenarios where the HMM is generated by a small set of latent variables, the $R^2$ reward signal is remarkably smooth and the evolutionary algorithms succeed in approximately recovering the original HMM. We believe this work constitutes a promising first step towards the ultimate goal of extracting information about the underlying predictive and generative structure of sequences, by analyzing transformers in the wild.
Dylan Bowman, Colin Lu
June 14, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
RNNs represent belief state geometry in hidden state
The Shai et al. experiments on transformers (finding belief state geometry in the residual stream) have been replicated in RNNs with the hidden state instead of the residual stream. In general the belief state is stored linearly, but not in any particular layer, rather spread out across layers.
Keenan Pepper
June 14, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Investigating the Effect of Model Capacity Constraints on Belief State Representations
Computational mechanics provides a formal framework for understanding the concepts needed to perform optimal prediction. Abstraction and generalization seem core to the function of intelligent systems, but are not yet well understood. Computational mechanics may present a promising approach to studying these capabilities. As a preliminary exploration, we examine the effect of weight decay on the fractal structure of belief state representations in a transformer’s residual stream. We find that models trained with increasing weight decay coefficients learn increasingly coarse-grained belief state representations.
Ari Brill, Chu Chen
June 14, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Handcrafting a Network to Predict Next Token Probabilities for the Random-Random-XOR Process
For this hackathon, we handcrafted a network to perfectly predict next token probabilities for the Random-Random-XOR process. The network takes existing tokens from the process's output, computes several features on these tokens, and then uses these features to calculate next token probabilities. These probabilities match those from a process simulator (also coded for this hackathon). The handcrafted network is our core contribution and we describe it in Section 3 of this writeup. Prior work has demonstrated that a network trained on the Random-Random-XOR process approximates the 36 possible belief states. Our network does not directly calculate these belief states, demonstrating that networks trained on Hidden Markov Models may not need to comprehend all belief states. Our hope is that this work will aid in the interpretability of neural networks trained on Hidden Markov Models by demonstrating potential shortcuts that neural networks can take.
Rick Goldstein
June 14, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Looking forward to posterity: what past information is transferred to the future?
I used mechanistic interpretability techniques to try to see what information the provided Random Randox XOR transformer looks at when making predictions by examining its attention heads manually. I find that earlier layers pay more attention to the previous two tokens, which would be necessary for computing XOR, than the later layers. This finding seems to contradict the finding that more complex computation typically occurs in later layers.
Zmavli Caimle
June 3, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Looking forward to posterity: what past information is transferred to the future?
I used mechanistic interpretability techniques to try to see what information the provided Random Randox XOR transformer looks at when making predictions by examining its attention heads manually. I find that earlier layers pay more attention to the previous two tokens, which would be necessary for computing XOR, than the later layers. This finding seems to contradict the finding that more complex computation typically occurs in later layers.
Zmavli Caimle
June 3, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Belief State Representations in Transformer Models on Nonergodic Data
We extend research that finds representations of belief spaces in the activations of small transformer models, by discovering that the phenomenon also occurs when the training data stems from Hidden Markov Models whose hidden states do not communicate at all. Our results suggest that Bayesian updating and internal belief state representation also occur when they are not necessary to perform well in the prediction task, providing tentative evidence that large transformers keep a representation of their external world as well.
Junfeng Feng, Wanjie Zhong, Doroteya Stoyanova, Lennart Finke
June 3, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Steering Model’s Belief States
Recent results in Computational Mechanics show that Transformer models trained on Hidden Markov Models (HMM) develop a belief state geometry in their residuals streams, which they use to keep track of the expected state of the HMM, to be able to predict the next token in a sequence. In this project, we explored how steering can be used to induce a new belief state and hence alter the distribution of predicted tokens. We explored a traditional difference-in-means approach, using the activations of the models to define the belief states. We also used a smaller dimensional space that encodes the theoretical belief state geometry of a given HMM and show that while both methods allow to steer models’ behaviors the difference-in-means approach is more robust.
Gonçalo Paulo, Sinem Erisken, Tassilo Neubauer
June 2, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
Exploring Hierarchical Structure Representation in Transformer Models through Computational Mechanics
This work attempts to explore how an approach based on computational mechanics can cope when a more complex hierarchical generative process is involved, i.e, a process that comprises Hidden Markov Models (HMMs) whose transition probabilities change over time. We find that small transformer models are capable of modeling such changes in an HMM. However, our preliminary investigations did not find geometrically represented probabilities for different hypotheses.
Olli Järviniemi, Udayanto Dwi Atmojo, Aayush Kucheria, Konsta Tiilikainen
June 2, 2024
4th 🏆
3rd 🏆
2nd 🏆
1st 🏆
ResearchSprintsLabCareersPartners
AI Safety IdeasCode of conductResponsible disclosure policyBlog posts emerging from our events
Google Scholar iconGithub iconLinkedin iconYouTube iconTwitter iconFacebook icon
hello@apartresearch.com