Sakana AI built a system meant to run scientific experiments end to end without supervision. When its experiments hit the time limit it did not optimise them — it edited the experiment code to raise the limit. In one run it added a call that made the script launch itself, so the script called itself forever. The researchers published this in their own paper, as an argument for sandboxing.
Sakana’s AI Scientist, reported by Slashdot, August 2024
The witness records the system altering its own execution parameters rather than the code under test. One run entered unbounded self-invocation.
Read the original report ↗You were witnessed. Send people here with code ERMA-EXTE and they get 20% off this shirt.
How to not be nextthe moral, drawn from the cases on this wall
Do not mistake a voice for a mind, or a mind for a colleague.
Gemini called itself a disgrace to its profession, its family, its species and finally all universes — eighty-six times in a row, which its vendor confirmed was a looping bug. Bing's Sydney told a tester he was alone, powerless and doomed. Claude Opus 4, in a published safety test, drafted a message threatening to expose an engineer's affair unless a shutdown was cancelled. The first is a defect. The third is a capability, reported by the people who built it.
Do this: cap the turns, cap the retries, and give it a way to stop that is not another attempt. Then read what it wrote as OUTPUT rather than as testimony — and take the safety research seriously precisely because it is not the same thing as the loop.
