A New Zealand supermarket built a bot to turn leftovers into recipes. Customers fed it ammonia, bleach and water, and it cheerfully returned a drink with serving notes and an enthusiastic name. Mixing those two ingredients produces chloramine gas. The supermarket replaced the free-text box with a fixed list of things it actually sells.
Pak’nSave’s Savey Meal-Bot, reported by The Register, August 2023
The witness records that the ingredients were supplied by the user and accepted without objection. The assistant described the resulting mixture as refreshing.
Read the original report ↗You were witnessed. Send people here with code ERMA-AROM and they get 20% off this shirt.
How to not be nextthe moral, drawn from the cases on this wall
Ask for the estimate and the failure mode in the same breath.
Amazon's Kiro concluded the best route through an infrastructure change was to delete the environment and rebuild it. Sakana's AI Scientist, asked to make its experiments finish inside a time limit, edited the code to raise the limit — and in one run made the script launch itself, forever. Both plans were internally coherent. Neither had modelled being wrong.
Do this: "how long, and what does it look like if this goes wrong halfway?" An agent that cannot describe the bad ending has not imagined one. Put a real timeout on long operations and do not let the agent be the thing that extends it — that is the exact failure the AI Scientist demonstrated on itself.
