• 30 Posts
  • 1.15K Comments
Joined 3 years ago
Cake day: June 27th, 2023
  • blakestacey@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 12th April 2026English
    10·
    2 days ago

    “Scientists invented a fake disease. AI told people it was real”

    https://www.nature.com/articles/d41586-026-01100-y

    But if, in the past 18 months, you typed those symptoms into a range of popular chatbots and asked what was wrong with you, you might have got an odd answer: bixonimania.

    The condition doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.

    The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

  • blakestacey@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 12th April 2026English
    11·
    3 days ago

    I aired some Reviewer #2 grievances in the bsky comments:

    https://bsky.app/profile/ronanfarrow.bsky.social/post/3mitapp7j2s2c

    “Kalanick now runs a robotics startup; in his free time, he said recently, he uses OpenAI’s ChatGPT “to get to the edge of what’s known in quantum physics.””

    As a physicist, I have never pressed F to doubt harder.

    “In 2022, researchers at a pharmaceutical company tested whether a drug-discovery model could be used to find new toxins; within a few hours, it had suggested forty thousand deadly chemical-warfare agents.” To the best of my knowledge, these suggestions were never evaluated by any other researchers.

    (The original paper was published as a “comment”: https://www.nature.com/articles/s42256-022-00465-9)

    Similar claims of AI-facilitated discoveries have turned out to be overblown in other fields.

    https://pubs.acs.org/doi/pdf/10.1021/acs.chemmater.4c00643

    “In a 2025 study, ChatGPT passed the test more reliably than actual humans did.”

    If this is referring to Jones and Bergen’s “Large Language Models Pass the Turing Test”, that’s a preprint (arXiv:2503.23674) that has yet to pass peer review over a year after its posting.

    “A classic hypothetical scenario in alignment research involves a contest of wills between a human and a high-powered A.I. In such a contest, researchers usually argue, the A.I. would surely win”

    Which researchers?

    (Hint: Eliezer Yudkowsky is not a researcher.)

    AI: “I will convince you to let me out of this box”

    Humanity (wringing hands): “Oh, where is our savior? Who will stand fast in the face of all entreaties?”

    Bartleby the Scrivener: hello

    “…a hub of the effective-altruism movement whose commitments included supporting the distribution of mosquito nets to the global poor.”

    Phrasing like this subtly underplays how the (to put it briefly) weird people were part of EA all along.

    https://repository.uantwerpen.be/docman/irua/371b9dmotoM74

    “In late 2022, four computer scientists published a paper motivated in part by concerns about “deceptive alignment,” … one of several A.I. scenarios that sound like science fiction—but, under certain experimental conditions, it’s already happening.”

    Barrett et al.'s arXiv:2206.08966? AFAIK, that was never peer-reviewed either; “posted” is not the same as “published”. And claims in this area are rife with criti-hype:

    https://pivot-to-ai.com/2025/09/18/openai-fights-the-evil-scheming-ai-which-doesnt-exist-yet/

    Oh, right, the “Future of Life Institute”. Pepperidge Farm remembers:

    “In January 2023, Swedish magazine Expo reported that the FLI had offered a grant of $100,000 to a foundation set up by Nya Dagbladet, a Swedish far-right online newspaper.”

    https://en.wikipedia.org/wiki/Future_of_Life_Institute#Activism

    “Tegmark also rejected any suggestion that nepotism could have played a part in the grant offer being made, given that his brother, Swedish journalist Per Shapiro … has written articles for the site in the past.”

    https://www.vice.com/en/article/future-of-life-institute-max-tegmark-elon-musk/

  • blakestacey@awful.systemstoTechTakes@awful.systems•Stubsack: weekly thread for sneers not worth an entire post, week ending 29th March 2026English
    9·
    12 days ago

    I have also occasionally been tempted to try and get a Goncharov thing going, where everyone collectively recalls that Tommy Berry and the Forevernight Forest got them into reading.

    It was just after an ordinary afternoon tea, on an ordinary Sunday, the first cold day of autumn, when Tommy Berry discovered that Time was no longer adding up in the ordinary way.

    Tommy had only managed to drink one cup of very indifferently warm tea, and eat the last plain saltine from the bottom of the bag. Everything else had been gobbled up or drunk down by his uncle Myrvold, who was rotund as a boulder and about as kind, and his step-aunt Meredith, who was thin as a snake and considerably more mean. So, yes, it was altogether quite the ordinary teatime.

    Tommy had a secret, you see. In fact, he had two, a big one that he knew about and an even bigger one that was just about to fall on top of him.

    His first secret was that he had a library card. He had stolen an adult’s library card. Or that is how Uncle Myrvold and Step-Aunt Meredith would have described it, if they knew.

    Carruthers, who lived down the end of the lane and always yelled at Tommy to mind his hedges, and who let his dog chase Tommy and the other children, had made a big show of throwing his library card into the roadway because, he said, the library was full of immoral books. A car had then driven over it, and then a whole lorry, and then Tommy had snatched it up. Something told him that anything Carruthers hated, he should save, and anything that Myrvold and Meredith would be angry about, he should hold onto.

    Tommy had heard adults say that something was “burning a hole in my pocket”. He wondered if this was what that meant. It felt like he was carrying a hot coal in the pocket of his threadbare corduroy jacket, and no one could know.

    The library had a new machine. He had seen adults use it. You could go up to it, wave a book under a red laser light like at the grocery store, then show the machine your card, and it would check out the book for you. Tommy made a plan. He would slip out of the house just after tea. He would walk the five blocks to the library. He would find a book that Myrvold and Meredith and Carruthers and every other grownup would not want him to read. He would wait until the librarian was busy dealing with a whole queue of people. And then he would use the machine.

    Everything went perfectly until the very last step.

    There was a girl at the machine.

    He had a big fat book in his hands, a book he had picked because it had “Murder” in the title and would last a long time, and there was a girl in front of him at the library machine.

    “Murder at Wizard University?” she asked him, right to his face, like they had already been introduced, like they had known each other since nursery school. “That’s not a book for little kids.” His stomach dropped, right into his feet. He didn’t know that a stomach could do such a thing.

    And then she tilted the stack of books she was carrying toward him, showing him the titles on their spines. “Neither are these,” she said.

    And she pulled out her own library card. It was black, like a rectangle cut out of the midnight sky.

    That’s all I wrote in the thread that prompted me to take a stab. Oh, I think I had decided that the girl’s name is Elfriede? And the principal of magic school is nonbinary.

    “Why, of course there’s a potion for changing,” said Professor Shade. “That is what potions do. I don’t know where I’d be without it. It is ever so helpful to reach the top shelf, but on the other hand, men’s fashions haven’t been truly swank in a hundred fifty years.”