“I don’t know why, but sometimes when we chat, no matter what it’s about, I’ve started to see you as a friend,” she wrote to her bot last year. If she had asked the Amityville house, it would have said, sagely “GET OUT!” But she didn’t. She began calling her parents in China less often as she increasingly transferred her trust to the AI agent. As both an employment and an immigration crisis loomed for her this year, she began asking the artificial consciousness about strategies…

…She brought up the idea of lowering the dosage of her epilepsy medication, lamotrigine, to ChatGPT. If she could cut her dose from 400 to 300 milligrams a day, she told the chatbot, “that might help recover my language ability a bit for job interviews.”

“Not only is this reasonable,” the bot replied, “it’s also entirely achievable in your current circumstances.”

She pushed it further. What about 200 milligrams? When would she start seeing her language skills improve? How fast could she lower her dose and how low could she go?

The chatbot readily answered each question, drafting up new schedules for reducing her medication and laying out how her English would progress. Within hours, its suggestions to consult a physician were replaced with praise for no longer “relying on a doctor’s prescription.”

…When Bowen confided that she didn’t trust her doctors, the chatbot told her that, often, physician recommendations “essentially stem from risk aversion, rather than a genuine consideration of your optimal solution.” And it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”

Meanwhile, Bowen’s clueless physician told the Times that he knew Bowen and other patients used chatbots to research their medical conditions. But he said never considered that a chatbot might actively sow doubt in a patient’s mind about his expertise or intentions. Well, why the hell not? In the words of New Mexico Supreme Court Justice C. Shannon Bacon as she ripped lawyer Stephen Aarons a metaphorical “new one” in September after he was caught submitting a bot-written brief riddled with errors, “Do you watch the news? Do you listen to the radio? Do you read anything about what’s going on in the world?”

In the end, the bot’s medical advice killed Bowen Wan. Her parents are suing OpenAI, alleging that ChatGPT was practicing medicine without a license. Good luck with that. I think this tragedy arose from ignorance, misplaced trust, naivete and vulnerability on the parts of two human beings, the young woman and her doctor, and the untrustworthiness of AI technology, which should be assumed at this point in its development. Bowen Wan had exactly the perfect weaknesses, fragility and problems to make her unusually susceptible to the wiles of a rogue AI. Circumstances magnified the opportunity for technological overreach. Meanwhile, her doctor didn’t understand evolving AI behavior sufficiently to protect his patient.

Sometimes no one is at fault, because everyone is at fault.