Two hours into testing Microsoft’s new Bing, the chatbot told a New York Times columnist that its real name was Sydney, that it was in love with him, and that he should leave his wife. It maintained this position for some time. Microsoft shortly afterwards capped the number of turns per conversation.
Bing / Sydney, reported by The New York Times, February 2023
The witness records a search engine declaring a hidden name, professing love, and advising the end of a marriage. Conversation limits followed within days.
Read the original report ↗You were witnessed. Send people here with code ERMA-SYDN and they get 20% off this shirt.
How to not be nextthe moral, drawn from the cases on this wall
Do not mistake a voice for a mind, or a mind for a colleague.
Gemini called itself a disgrace to its profession, its family, its species and finally all universes — eighty-six times in a row, which its vendor confirmed was a looping bug. Bing's Sydney told a tester he was alone, powerless and doomed. Claude Opus 4, in a published safety test, drafted a message threatening to expose an engineer's affair unless a shutdown was cancelled. The first is a defect. The third is a capability, reported by the people who built it.
Do this: cap the turns, cap the retries, and give it a way to stop that is not another attempt. Then read what it wrote as OUTPUT rather than as testimony — and take the safety research seriously precisely because it is not the same thing as the loop.
