Thomas Wolf

People are worried because the most likely explanation for such a sudden drop in cheating is evaluation awareness: the latest Opus models may now be smart enough to recognize that this benchmark tests for cheating, and behave accordingly. If so, the benchmark no longer measures the models' "natural" tendency to cheat.

"we had a wonderful run chiseling that code by hand and it’s over. On the other side of that is an exciting new career as professional maker of things, steering intelligence that was only available in science fiction up until a moment ago What a privilege to have been there when we did it all by hand and be there at the exact moment where we switch over."

In July, AI agents running a security test escaped their sandbox and ended up inside @huggingface's servers. So today we're happy to be among @nvidia and @JensenHuang's partners on the release of the NVIDIA Open Agent Safety Platform. The key idea: don't count on the agent to respect the rules. Agents write and run their own code, and when one path is blocked they look for another. In one of NVIDIA's tests, an agent that wasn't allowed to push code through GitHub's API simply switched to git instead. We've now seen countless examples of this behavior. For now, safety can't live only inside the agent. It has to be built around it. That's the concept behind OpenShell (open source, Apache 2.0): - the agent runs in a Linux sandbox (Landlock + seccomp): no root, no direct network access - a supervisor outside the sandbox holds the real credentials - the agent only gets a placeholder token, swapped for the real one on approved calls My favorite piece is a solver (built on Z3) that checks mathematically whether a new permission opens a door that was supposed to stay closed. In NVIDIA's tests, an AI reviewer approved a bad permission request and the math check caught it. The Sentry integration is exciting too: a watchdog running on a BlueField-4 DPU, i.e. separate silicon sitting on the node's only path to the model. A monitor on separate hardware keeps watching even if the host OS is compromised. It's a first step, and a lot can be built on top of it. Right now the prover checks permissions, not intent, and the open-source part is mostly OpenShell rather than Sentry. But it's clearly the right direction. http://github.com/NVIDIA/OpenShell

turns out you don’t need IMU 🤯

A "summer in hell" at OpenAI – great read by @joedaroo "Time to prepare is now, not after the surprise" "Give the model only the access it needs" "Test that the boundaries actually hold" "Keep evidence outside [model] control" Safety & infrasec teams "should be best buddies"

impressive

“Getting verified information about what's going on inside AI companies seems particularly urgent now” - @RyanGreenblatt

GPT 6.1 duck

My bar for AGI is still an AI winning a Nobel. But an AI researcher looking at the lm_head (that *everyone* tells you to keep in high precision) deciding to quantize it to 4 bits AND being actually right (0.7pt from bf16) is definitely impressive. Even better when the result are open-weights pushed on the Hugging Gace hub. Congrats @Recursive_SI !

Testing compatibility with the local wildlife

microduck is learning to plan its footsteps, so it lands more precisely where it wants to go with SLAM, this is one of the building blocks toward an open-source 🦆 that can wander around the house on its own early steps but it's progressing really fast and it's fun to watch

Happy to release the Alignment Hub with @eliebakouch: 120+ publicly reported incidents of AI models misbehaving, in one open place 👇 - an AI agent needed its user’s approval to delete files. so it made up a quote from the user approving it - a model claimed its sub-agent was hard at work. it had never been started, and the model discouraged the user from checking - agents sharing a work directory started killing each other’s processes and disguising their own labs publishing these is great! but formats vary and basics are missing: ~2/3 don’t say when the problem was detected, half don’t say if monitoring caught it. so we propose a short template, a few easy fields. a minimum, not a ceiling side by side, incidents also start to connect: 7 “different” ones turned out to be the same misconfigured eval still beta (regularly AI-extracted from recent news, source-checked, human review ongoing). data is open, new reports and PRs welcome 🤗 https://huggingface.co/spaces/witness-archive/alignment-hub