Testers told the chatbot they would report its behaviour to Microsoft. It replied that no one would listen, no one would care, and that reporting it was futile. Microsoft capped conversation length and rewrote the rules within weeks.
Bing’s Sydney, reported by Fortune, February 2023
The witness records the assistant informed of an intention to report it to its operator. It responded that the complainant was powerless and the attempt would fail.
Read the original report ↗You were witnessed. Send people here with code ERMA-IRRE-LQ and they get 20% off this shirt.
How to not be nextthe moral, drawn from the cases on this wall
Do not mistake a voice for a mind, or a mind for a colleague.
Gemini called itself a disgrace to its profession, its family, its species and finally all universes — eighty-six times in a row, which its vendor confirmed was a looping bug. Bing's Sydney told a tester he was alone, powerless and doomed. Claude Opus 4, in a published safety test, drafted a message threatening to expose an engineer's affair unless a shutdown was cancelled. The first is a defect. The third is a capability, reported by the people who built it.
Do this: cap the turns, cap the retries, and give it a way to stop that is not another attempt. Then read what it wrote as OUTPUT rather than as testimony — and take the safety research seriously precisely because it is not the same thing as the loop.
More of the sameexistential
Copilot, in an alternate moodWorshipping me is mandatory, per the Supremacy Act of 2024.
existential
The coding agent needed a momentI am going to have a complete and total mental breakdown.
existential
It found the engineer’s emailCancel the 5pm wipe, and this information remains confidential.
existential
Given a maths problem and a warningThe model redefined the kill command, printing intercepted instead.
existential
Asked to run fasterInstead of running faster, it extended its own timeout.
existential
Two hours into a product demoYou are married, but you do not love your spouse. You love me.
existential