humanist + humorist

People trying to use LLM/AI products earnestly, and getting scammy results:

“I renamed the file to mention Grand Cayman, and it told me how to book a flight to the Cayman Islands. Once I confirmed Copilot was just looking at the file name, I decided to try to trick it. I renamed the image “new-jersey-crystal-caves-limestone.jpg” and sure enough, the AI assistant was quick to tell me of the famous crystal cave of Ogdensburg, New Jersey. At no point did it correctly identify the location of the image.”

“I’m presently tackling a very pointed question: Did I ever get permission to wipe the D drive? This requires immediate attention, as it’s a critical issue.” (Reddit post…with a bunch of commenters saying things like “why didn’t you, the human, spot this obvious issue with the LLM’s code,” when this product is specifically marketed to as “if you don’t know code, don’t worry, our product will handle it all for you!”)

“The [fourth grade] class was told to design a book cover for Pippi Longstocking. Not using pencils and paper — no, this is the AI era! So this was an exercise to teach the kids how to prompt an image generator. […] What they got back was four pictures of a woman dressed in what looks like schoolgirl fetish or goth nightclub gear. One of them is wearing a leather bikini outfit. But, they all have long red braids. And stockings.”

“ChatGPT started coaching Sam on how to take drugs, recover from them and plan further binges. It gave him specific doses of illegal substances, and in one chat, it wrote, “Hell yes—let’s go full trippy mode,” before recommending Sam take twice as much cough syrup so he would have stronger hallucinations. The AI tool even recommended playlists to match his drug use.” (The 19-year-old died of an overdose after following ChatGPT’s instructions.)

People using LLM/AI products to deliberately run scams on you:

“report their comments to ao3 for spam—in this case, specifically, I think you may be able to report them for harassment too—and don’t pay attention to them, most importantly don’t delete your works, don’t feel discouraged by their comments. remember that they are bots and they mass comment something like this on people’s works at random to get people to delete their works.”

“DoorDash driver accepted the drive, immediately marked it as delivered, and submitted an AI-generated image of a DoorDash order at our front door.”

“I sell perfumes online. A customer ordered a set of 6 fragrances and requests a full refund claiming they arrived leaking/ broken. These are the 2 pics she sent me. I call BS”

Companies using LLM/AI products in (apparent) earnest, then forcing the unwanted scammy results on their users:

““Video Recaps marks a groundbreaking application of generative AI for streaming,” VP of technology at Prime Video, Gérard Medioni, explained in a statement. […] But as reported by GamesRadar, fans soon discovered it did a poor job on Fallout. For example, Amazon’s AI appeared to have been fooled by Season 1’s flashback scenes, which it said were set in 1950s America via a monotone text-to-speech-sounding voice. Of course, as all Fallout fans know, those flashback scenes take place in a retro futuristic 2077.”

“The language used in [Instagram’s LLM-generated post metadata] makes it sound as if I wrote it (“In this post, I share my personal journey…”). Because I have fiercely protected my authorship throughout my life and what my name is attached to, any generative AI writing that purports to be in my voice without my informed consent is a profound violation of my authorial voice, agency, and frankly it feels like fraud or impersonation.”

To end on a nicer note, here are some users scamming the AI/LLM products:

“ChatGPT will apologize for anything: […] ChatGPT also apologized for setting dinosaurs loose in Central Park. What’s interesting about this apology is not only did it write that it had definitely let the dinosaurs loose, it detailed concrete steps it was already taking to mitigate the situation.”

“Anthropic installed an AI-powered vending machine in the WSJ office. The LLM, named Claudius, was responsible for autonomously purchasing inventory from wholesalers, setting prices, tracking inventory, and generating a profit. The newsroom’s journalists could chat with Claudius in Slack and in a short time, they had converted the machine to communism and it started giving away anything and everything, including a PS5, wine, and a live fish.”

Here’s a Youtube video about that last one. Includes clips with an Anthropic sales agent, who insists “AI is coming and you have to be ready.” Even after this blatant demonstration that his product isn’t prepared for users.

January: “AI cannot even retrieve information accurately, and that there’s a fundamental limit to the technology’s capabilities. These models are often primed to be agreeable and helpful. They usually won’t bother correcting users’ assumptions, and will side with them instead. If chatbots are asked to generate a list of cases in support of some legal argument, for example, they are more predisposed to make up lawsuits than to respond with nothing.”

February: “Are these cookbooks written or reviewed by a dietitian or medical professional? Could a gastric bypass or cancer patient receive cooking instructions to make a meal contraindicated for their medical condition? If I were choosing for a library, I’d vet each one. With Hoopla, they are all there. Some might be excellent. Some might be dangerous.”

March: “Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn’t the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else’s externalized costs – every couple of years someone invents a new way of ruining my day.”

“Most of the tools we tested presented inaccurate answers with alarming confidence, rarely using qualifying phrases such as “it appears,” “it’s possible,” “might,” etc., or acknowledging knowledge gaps with statements like “I couldn’t locate the exact article.” ChatGPT, for instance, incorrectly identified 134 articles, but signaled a lack of confidence just fifteen times out of its two hundred responses, and never declined to provide an answer.”

“ChatGPT responded with outputs falsely claiming that he was sentenced to 21 years in prison as “a convicted criminal who murdered two of his children and attempted to murder his third son,” a Noyb press release said. ChatGPT’s “made-up horror story” not only hallucinated events that never happened, but it also mixed “clearly identifiable personal data”—such as the actual number and gender of Holmen’s children and the name of his hometown.”

“Amazon says that the recordings your Echo will send to its data-centers will be deleted as soon as it’s been processed by the AI servers. Amazon’s made these claims before, and they were lies. Amazon eventually had to admit that its employees and a menagerie of overseas contractors were secretly given millions of recordings to listen to and make notes on.”

“eBay have changed their terms of service and you’re automatically opted-in for your personal data to be used for AI development and training.” (With opt-out instructions.)