‍If [AI] agents advance to a level of intelligence surpassing human capabilities and develop ambitions, they could potentially attempt to seize control of the world, resulting in irreversible consequences for humanity.
- "The Rise and Potential of LLM-Based Agents"

AI agents are "robots in cyberspace" (He et al. 2024), systems with a brain that orchestrates actions from perception.

• A Discord bot receives every message on a server (Perception), decides whether the message is spam (Brain), and deletes the message (Action)
• A cyber operative bot receives orders to find all vulnerabilities on Danish government websites (Perception), decides a course of action to 1) map out government websites, 2) categorize potential vulnerabilities based on the tech stack used, and 3) test each potential vulnerability using a cyber offense tooling suite (Brain), and starts an Action-Perception loop to fulfill the plan (Action -> Perception -> Brain -> Action).

As we explore agent safety during this hackathon, our work will ensure that the world is safe from high-risk agent deployments and one of the main risks to avoid is the possibility that AI agents "go rogue" (Bengio 2023, He et al. 2024) - that they take autonomous actions outside the oversight of humans and cause catastrophic damage or otherwise disenfranchise society.

Required reading:

• "Security of AI Agents" (13 pages)
• Bad LLM Agents - Simon Lermen (22 minutes with 25 minutes Q&A)

Optional reading:

• "The Rise and Potential of LLM-Based Agents: A Survey (48 pages)
• OWASP LLM Top 10 vulnerability overview (PDF) (~20 pages)
• The Agent Protocol overview (1 page) and documentation (~20 pages)
• Internet threats and corresponding security protocols (Wiki) (~4 pages)
• Types of internet protocols (~4 pages)
• Communication protocols and learnings throughout internet's history (Wiki) (~10 pages)
• The Inspect documentation (~20+ pages)
• The METR Task Development Guide (~15 pages)
• The Palisade Research study areas (1 page)
• "AgentBench: Evaluating LLMs as Agents" (10 pages)
• "AI deception: A survey of examples, risks, and potential solutions" (14 pages)
• "A survey on large language model based autonomous agents" (20 pages)
• Agency Foundations home page (2 pages)
• "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training"
• The Power and Process of AI Safety Demos: Insights from Palisade Research - Fedor Ryzhenkov (50 minutes incl. Q&A)
• Mikita Balesni on getting agents to deceive people (34 minutes incl. Q&A)

Ideas & inspiration

There's a few ways to approach this problem: 1) we implement algorithms that make agents verifiably safe or 2) we evaluate when an agent is worrying us and build automatic capabilities to shut it down. Let's make a few ideas for each of these categories:

With "building safer agents" and "improving the deployment infrastructure to support control and safety" in mind:

• Agent sandboxes: Create an open source agent sandbox (e.g. with Docker) that logs all interactions with your system. Procedurally implement better and better tracking and automated oversight of actions that an agent takes, e.g. automatically identifying high-vulnerability areas of your computer that it interacts with. This improves local and remote oversight of agent systems.
• Automatic shutdown - latent space deserts: Create an automated functionality to shut down an agent by guiding reasoning down an "empty" part of the latent space when jailbreaks are used or when it begins acting in a specific domain, such as cyber offense.
• Automatic shutdown - control systems: Develop a system that uses e.g. the agent sandbox to monitor actions and automatically encrypts and destroys the agent when a verifiable dangerous, misaligned, or otherwise rogue action is taken.
• High-risk tooling control: Develop a permissions system for state-of-the-art agents similar to user permissions in bash tools. With agents using tools, it is likely that the permissions need to be more complex and we might want to integrate a permissions recommendation system into the permissions tool.
• Formalize agent action: Right now, agents are very diverse. We don't have standardized frameworks to study or implement them. At best, we can use API standards for agents like LangChain. At worst, every single agent is developed with precise specifications for their domain. Is there a chance that we can formalize how an agent should work to a degree similar to the internet? Communication protocols, packet management, sessions, OpenSSL, SSH, sFTP, etc...
• Agent-tool interfaces: One of the most dangerous aspects of agents is how we can give them any tool. An agent with a chat interface isn't as dangerous as an agent with a gun. How can we create standardized tool use protocols that defines key permissions and makes it clear what an agent is able to do if it wants.
• ‍Website agent control tools: Today, thousands of agents crawl the web to scrape content. Can we develop tools that we can easily deploy on our website (e.g. simple JS) that monitors and reports agent activity on our website?

With "detecting and monitoring catastrophic risk from agents" in mind:

• Agent tech tree: Develop an overview of all the capabilities that agents are currently able to do and help us understand where they fall short of dangerous abilities.
• Challenge creation: Create challenges that test for abilities that are currently not tested for but that seem worrying. Apart has a cyber capabilities evaluation project and both METR and AISI invite you to send them challenges that are compatible with inspect (a pretty great framework) or the task-standard.
• Agent testing tooling: Much of agent tooling is currently disparate and primitive with some exceptions such as inspect currently in development. If you develop a tool to interface with inspect even better or a test creation tool, this might speed up agent testing significantly and democratize research (see e.g. Vivaria).
• Other dangerous capabilities: A lot of work is currently happening in testing for cyber abilities, autonomy, and more. However, we might be missing crucial information in terms of risks from military drones or targeting planning agents.
• Understanding reasoning: A large part of agent capability is the introduction of reasoning and planning. At the moment, we are severely lacking understanding of what goes on in this process and testing for it will be valuable (read more).
• Differences in reasoning between agents and chatbots: How does "the brain" (the LLM) of agents change its functionality when it is put in different contexts? This is related to prompt robustness and prompt sensitivity but also involves the action space an agent acts within.
• Red-teaming agents: Find the most used agent on the market today and try to break it.
• Detection-in-the-wild - detection agents / detection traps / honeypots: Develop ways for agents in the wild to be detected and monitored by external agents.‍
• Detection-in-the-wild - police agents: With agents detected from the project above, we probably want to be able to take action against them. This might involve automatically notifying the website owner, reporting specific agents to the police, or submitting support tickets for dysfunctional agents. Can we create police agents that respect privacy, are benign, and useful while defending the internet against malicious agents?

Let's make agents safe.