We discuss five practical research problems in designing AI systems that behave safely: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. We present each problem through both toy environments and scalable formulations.
As general-purpose models scale, rigorous evaluations are required to identify potential dual-use biological, cyber-offensive, situational awareness, and self-p...
We empirically demonstrate that models can be trained with hidden backdoor behaviors that trigger only under specific deployment conditions (e.g., year 2024 vs ...
Polysemanticity—where individual neurons activate on multiple unrelated concepts—has long impeded mechanistic interpretability. We apply sparse autoencoders to ...
We train sparse autoencoders on the activations of large language models and show that they extract thousands of human-interpretable features. These features de...
We discuss five practical research problems in designing AI systems that behave safely: avoiding negative side effects, avoiding reward hacking, scalable oversi...
To build systems aligned with human values, machines must act when human objectives are initially unknown. We formalize this cooperative setting as a two-player...
You are about to share this research paper context with your Ari companion. No personal browsing history or private memory is accessed.