“If the LLM produces a wild result, something that doesn’t meet with my expectations *then* I’ll turn to more reliable sources.
-
@futurebird
3/3
Google can weigh a source for their own LLM, making it insist on a thing, but they can't weigh their own sources for credibility, other than frequency in the training data.So, the most commonly held beliefs are automatically true and will be referenced as such by an LLM.
It's a confirmation bias machine for all of humanity.
The end of Science.
@futurebird
4/3
Make no mistake, the fact that I could have a conversation like this with a machine is a great accomplishment. Turning a huge data set and some language rules into a thing I could query for hours is astonishing.But, current AI has a credibility problem, and that means that it is not ready as a truth telling product. And the hype outweighs the truthiness by an uncomfortable margin.
-
@futurebird
4/3
Make no mistake, the fact that I could have a conversation like this with a machine is a great accomplishment. Turning a huge data set and some language rules into a thing I could query for hours is astonishing.But, current AI has a credibility problem, and that means that it is not ready as a truth telling product. And the hype outweighs the truthiness by an uncomfortable margin.
There are a lot of of people I could talk to for hours and still go away being no wiser or better informed than I was before, but at least I was talking to a real person.
-
@futurebird
4/3
Make no mistake, the fact that I could have a conversation like this with a machine is a great accomplishment. Turning a huge data set and some language rules into a thing I could query for hours is astonishing.But, current AI has a credibility problem, and that means that it is not ready as a truth telling product. And the hype outweighs the truthiness by an uncomfortable margin.
@futurebird
A/C
Another experiment:I know of a data base that was populated by an early AI that 'hallucinated' details. An international kite museum, supposedly in Corpus Christi, Texas, was said to be populated by displays on "The Age of Mammoths" and "The Iron Horse" because the word "museum" took more precedence than "International Kite".
It hallucinated a lot of other generic museum like details.
A street view search of the address shows a hotel, and no museum at all for blocks around.
-
The most exciting and pivotal moments in research are those times when the results do not meet your expectations.
We live for those moments.
If an LLM is not reliable enough for you to trust unexpected results then it is not reliable enough to tell you anything new: it’s incapable of telling you anything that you don’t at (some level) already know.
2/2
@futurebird not to take away from anything you've said, it's definitely true that this sort of approach leads to dismissing novel ideas. but i think there's also another level at which this kind of approach is worrying: by saying they need to sometimes dismiss the machine's output, one admits the machine is not to be trusted, yet they trust it when they can't catch it in what's (seemingly) an obvious lie.
in other words, it's trusting the liar's word unless you already know it to be false. it's trusting the allegorical wolf to take care of the sheep because if you ever see it attacking one you'd stop it.if you know you can't trust the machine sometimes, then you can't trust the machine. and if your system is then "well i'll just catch the mistakes", then either you only intend to use it in cases where you already know the right answer (in which case, why use it), or you believe you'll somehow figure out something's wrong when you can't tell what "correct" looks like.
it's saying "if it looks correct then it's correct". which is wrong both on the false negatives end and the false positive ends at the same time.
-
@futurebird
A/C
Another experiment:I know of a data base that was populated by an early AI that 'hallucinated' details. An international kite museum, supposedly in Corpus Christi, Texas, was said to be populated by displays on "The Age of Mammoths" and "The Iron Horse" because the word "museum" took more precedence than "International Kite".
It hallucinated a lot of other generic museum like details.
A street view search of the address shows a hotel, and no museum at all for blocks around.
I think when some people are presented with these kinds of errors they think “the LMM just made a factual mistake” they think with more data and “better software” this will not be a problem. They don’t see that what it is doing is *foundationally different* from what they are asking it to do.
That it has fallen to random CS educators, and people interested in these models to desperately try to impress upon the public the way they are being tricked makes me angry.
-
F myrmepropagandist shared this topic
-
There are a lot of of people I could talk to for hours and still go away being no wiser or better informed than I was before, but at least I was talking to a real person.
@the5thColumnist @Urban_Hermit
People, even people who have terrible mostly wrong ideas tend to have some guiding set of values. Even if you don’t learn much from their opinions you can learn about the philosophy that informs those opinions.
Asking an LLM *why* it said any fact or opinion is pointless. It will supply a response that sounds like a human justification but the real “reason” is always the same “it was the response you were most likely to accept as correct”
-
@futurebird not to take away from anything you've said, it's definitely true that this sort of approach leads to dismissing novel ideas. but i think there's also another level at which this kind of approach is worrying: by saying they need to sometimes dismiss the machine's output, one admits the machine is not to be trusted, yet they trust it when they can't catch it in what's (seemingly) an obvious lie.
in other words, it's trusting the liar's word unless you already know it to be false. it's trusting the allegorical wolf to take care of the sheep because if you ever see it attacking one you'd stop it.if you know you can't trust the machine sometimes, then you can't trust the machine. and if your system is then "well i'll just catch the mistakes", then either you only intend to use it in cases where you already know the right answer (in which case, why use it), or you believe you'll somehow figure out something's wrong when you can't tell what "correct" looks like.
it's saying "if it looks correct then it's correct". which is wrong both on the false negatives end and the false positive ends at the same time.
-
@the5thColumnist @Urban_Hermit
People, even people who have terrible mostly wrong ideas tend to have some guiding set of values. Even if you don’t learn much from their opinions you can learn about the philosophy that informs those opinions.
Asking an LLM *why* it said any fact or opinion is pointless. It will supply a response that sounds like a human justification but the real “reason” is always the same “it was the response you were most likely to accept as correct”
@the5thColumnist @Urban_Hermit
I’m trying to avoid loaded phrases like “bullshitting machine” I’ve had a lot of people roll their eyes and shut down on me because “well you just hate AI” as if this is a matter of personal taste.
In reality I struggle to see how I could find value in exposing my curiosity to a system with these limitations. I will insulate myself from those times when a simple obvious question brings me up short— it just seems really dangerous to me.
-
I think with topics where one isn’t an expert it can be more important to know what “most people” in your social circle *think* is true than it can be to know what is really true.
Knowing an iconoclastic truth, but not having the expertise to explain it to others isn’t very useful. Moreover without that expertise you will struggle to evaluate the validity of the unpopular opinion.
So, people reach for the popular option and hope it is correct.
-
I think with topics where one isn’t an expert it can be more important to know what “most people” in your social circle *think* is true than it can be to know what is really true.
Knowing an iconoclastic truth, but not having the expertise to explain it to others isn’t very useful. Moreover without that expertise you will struggle to evaluate the validity of the unpopular opinion.
So, people reach for the popular option and hope it is correct.
No one is enough of a polymath and no one has enough time to avoid trusting others. This isn’t really a bad thing, but we have to be open to the reality that there are some things that “everyone knows” that are simply wrong.
-
No one is enough of a polymath and no one has enough time to avoid trusting others. This isn’t really a bad thing, but we have to be open to the reality that there are some things that “everyone knows” that are simply wrong.
@futurebird @alessandro @talya Drawing a line to my favourite topic:
Virtually everything we "knew" about the cognition and social and emotional lives of reptiles before the last couple of decades was wrong (due to systemically flawed studies, and old half-truths that ended up getting quoted enough to take on a weird life on their own). Many of those wrong things are still things that "everyone knows", and they're almost certainly well-represented in a training corpus that both includes older textbooks and all of Reddit. It's completely hit-or-miss if LLMs (at least the ones I've tried) answer with some long-debunked piece of archaic trivia or something well-established backed up by a modern herpetology article. (And when they say something wrong, they're often so authoritative and frequently back it up with consistent-but-wrong "reasoning" that I have to double-check to find out if *I* am wrong).
If we'd had enough computer power and data to train LLMs in 1995, virtually every single thing they would have had to say on the topic would be wrong!
(Making a judgment on whether everything we know in 2025 is so right that something like that will never happen again in any field is left as an exercise to the reader.)
-
@the5thColumnist @Urban_Hermit
People, even people who have terrible mostly wrong ideas tend to have some guiding set of values. Even if you don’t learn much from their opinions you can learn about the philosophy that informs those opinions.
Asking an LLM *why* it said any fact or opinion is pointless. It will supply a response that sounds like a human justification but the real “reason” is always the same “it was the response you were most likely to accept as correct”
@futurebird @the5thColumnist @Urban_Hermit Why do you say this versus "most likely to be statistically correct"?
Regardless, that doesn't mean it's the right answer nor the right answer for the person asking it.
-
@futurebird @alessandro @talya Drawing a line to my favourite topic:
Virtually everything we "knew" about the cognition and social and emotional lives of reptiles before the last couple of decades was wrong (due to systemically flawed studies, and old half-truths that ended up getting quoted enough to take on a weird life on their own). Many of those wrong things are still things that "everyone knows", and they're almost certainly well-represented in a training corpus that both includes older textbooks and all of Reddit. It's completely hit-or-miss if LLMs (at least the ones I've tried) answer with some long-debunked piece of archaic trivia or something well-established backed up by a modern herpetology article. (And when they say something wrong, they're often so authoritative and frequently back it up with consistent-but-wrong "reasoning" that I have to double-check to find out if *I* am wrong).
If we'd had enough computer power and data to train LLMs in 1995, virtually every single thing they would have had to say on the topic would be wrong!
(Making a judgment on whether everything we know in 2025 is so right that something like that will never happen again in any field is left as an exercise to the reader.)
@datarama @futurebird @alessandro @talya
Could you give an example, please?
I have to confess that I'm not sure if I've ever thought about how reptiles think. -
@futurebird @the5thColumnist @Urban_Hermit Why do you say this versus "most likely to be statistically correct"?
Regardless, that doesn't mean it's the right answer nor the right answer for the person asking it.
@elight @the5thColumnist @Urban_Hermit
Because the popular models people will encounter have been trained to work this way.
-
@datarama @futurebird @alessandro @talya
Could you give an example, please?
I have to confess that I'm not sure if I've ever thought about how reptiles think.@Gorfram @futurebird @alessandro @talya I just answered myself with a collection of things Copilot 95 would have gotten wrong.
The short version is that reptiles are neither particularly stupid, particularly asocial or particularly emotionless. In the 60s it was commonly believed that there was a primitive "reptile complex" in the brain that only covered the most basic instinct-driven behaviour and processing of simple stimuli. This is *still* part of popular understanding; it's what people mean when they talk about "the lizard brain". The concept got popularized by (rest his soul) Carl Sagan in his book "The Dragons of Eden" from 1977.
But it turns out an actual lizard brain is nothing like that primitive extended-brainstem structure he described! Lizards have a limbic system just like mammals (and they *definitely* experience emotion), some even practice parental care (in monkeytail skinks for up to a year), live in stable family groups and adopt orphans, and some others form lifelong monogamous pairs (shingleback skinks are famous for this; they even have a range of behaviour that is hard to explain as anything other than a kind of grief when a partner dies).
It used to be commonly accepted that reptiles were incapable of learning novel problem-solving behaviour, because they don't have a neocortex (which is what mammals use to do that). But it turns out that the dorsal ventricular ridge (which reptiles share with birds) plays a similar role to the mammalian neocortex, and packs an enormous amount of neurons into a comparatively tiny volume. Until 1997, we thought it was just a ganglion that handled motor control.
Little anoles with brains the size of pinheads do better on the worm-in-a-lid puzzle than most songbirds and can even learn by looking at other anoles. Monitor lizards can count to 6, tortoises can navigate mazes using something that resembles depth-first-search, crocodilians can put twigs and other items on their heads to use as lures for birds - even adapting which kinds of material they use according to which birds are currently building nests. Many can differentiate individual conspecifics *and* tell the difference between individual humans.
-
@Gorfram @futurebird @alessandro @talya I just answered myself with a collection of things Copilot 95 would have gotten wrong.
The short version is that reptiles are neither particularly stupid, particularly asocial or particularly emotionless. In the 60s it was commonly believed that there was a primitive "reptile complex" in the brain that only covered the most basic instinct-driven behaviour and processing of simple stimuli. This is *still* part of popular understanding; it's what people mean when they talk about "the lizard brain". The concept got popularized by (rest his soul) Carl Sagan in his book "The Dragons of Eden" from 1977.
But it turns out an actual lizard brain is nothing like that primitive extended-brainstem structure he described! Lizards have a limbic system just like mammals (and they *definitely* experience emotion), some even practice parental care (in monkeytail skinks for up to a year), live in stable family groups and adopt orphans, and some others form lifelong monogamous pairs (shingleback skinks are famous for this; they even have a range of behaviour that is hard to explain as anything other than a kind of grief when a partner dies).
It used to be commonly accepted that reptiles were incapable of learning novel problem-solving behaviour, because they don't have a neocortex (which is what mammals use to do that). But it turns out that the dorsal ventricular ridge (which reptiles share with birds) plays a similar role to the mammalian neocortex, and packs an enormous amount of neurons into a comparatively tiny volume. Until 1997, we thought it was just a ganglion that handled motor control.
Little anoles with brains the size of pinheads do better on the worm-in-a-lid puzzle than most songbirds and can even learn by looking at other anoles. Monitor lizards can count to 6, tortoises can navigate mazes using something that resembles depth-first-search, crocodilians can put twigs and other items on their heads to use as lures for birds - even adapting which kinds of material they use according to which birds are currently building nests. Many can differentiate individual conspecifics *and* tell the difference between individual humans.
@Gorfram @futurebird @alessandro @talya Oh - also, it turns out that many species of lizard have REM sleep and corresponding brainwave patterns. This means they almost certainly dream - something that was believed to only occur in mammals and birds until ... 2016 (where it was experimentally confirmed in bearded dragons)! That's about the same time Google started developing the very first Transformer-based LLM, in fact (the paper that introduced them came out in 2017).
This threw a lot of things we thought we knew out of whack. We *used* to think REM sleep was something specific to endotherms, and there were a lot of proposed explanations about why. It seemed like it'd have to be an evolutionarily recent adaptation. Perhaps something to do with the higher metabolism of endotherms? Perhaps only endotherms supplied enough energy to the brain during sleep for dreams to occur? All of those explanations made sense and seemed perfectly plausible, but they were also all wrong. Now we don't even know if some ancestral amniote whose descendants would branch off into distinct reptile and mammal lineages was dreaming when it slept some 320 million years ago, or if REM sleep evolved separately in the two lineages.
So there's not really any way Copilot 95 could have gotten this one right.
(...so why should I trust 2025's ChatGPT or Gemini to get it right about, say, amphibians? We haven't seen any evidence of REM sleep in them - but in 2015, we hadn't seen it in reptiles yet either. And just a couple years ago, we found something resembling REM sleep in some species of *spiders*, of all animals! Was some pre-Cambrian proto-animal dreaming too? Until aforementioned spider discovery we thought it'd be such an odd thing to evolve separately that we figured early amniotes "invented" REM sleep; now that's thrown into doubt as well.)
-
@Gorfram @futurebird @alessandro @talya Oh - also, it turns out that many species of lizard have REM sleep and corresponding brainwave patterns. This means they almost certainly dream - something that was believed to only occur in mammals and birds until ... 2016 (where it was experimentally confirmed in bearded dragons)! That's about the same time Google started developing the very first Transformer-based LLM, in fact (the paper that introduced them came out in 2017).
This threw a lot of things we thought we knew out of whack. We *used* to think REM sleep was something specific to endotherms, and there were a lot of proposed explanations about why. It seemed like it'd have to be an evolutionarily recent adaptation. Perhaps something to do with the higher metabolism of endotherms? Perhaps only endotherms supplied enough energy to the brain during sleep for dreams to occur? All of those explanations made sense and seemed perfectly plausible, but they were also all wrong. Now we don't even know if some ancestral amniote whose descendants would branch off into distinct reptile and mammal lineages was dreaming when it slept some 320 million years ago, or if REM sleep evolved separately in the two lineages.
So there's not really any way Copilot 95 could have gotten this one right.
(...so why should I trust 2025's ChatGPT or Gemini to get it right about, say, amphibians? We haven't seen any evidence of REM sleep in them - but in 2015, we hadn't seen it in reptiles yet either. And just a couple years ago, we found something resembling REM sleep in some species of *spiders*, of all animals! Was some pre-Cambrian proto-animal dreaming too? Until aforementioned spider discovery we thought it'd be such an odd thing to evolve separately that we figured early amniotes "invented" REM sleep; now that's thrown into doubt as well.)
@datarama @futurebird @alessandro @talya
Now I'm wondering what lizards & spiders dream about (I'm guessing flies figure pretty heavily).So many things in science were right until they were wrong. I remember being taught that the earth's crust rested on a generally quiescent layer of molten lava, which occasionally broke through at weak spots in the form of erupting volcanoes (plate tectonics was known in academic circles by then, but hadn't made it as far as our 1960's-era 4th-grade earth science textbooks).
-
@datarama @futurebird @alessandro @talya
Now I'm wondering what lizards & spiders dream about (I'm guessing flies figure pretty heavily).So many things in science were right until they were wrong. I remember being taught that the earth's crust rested on a generally quiescent layer of molten lava, which occasionally broke through at weak spots in the form of erupting volcanoes (plate tectonics was known in academic circles by then, but hadn't made it as far as our 1960's-era 4th-grade earth science textbooks).
@Gorfram @datarama @alessandro @talya
Ants and bees probably dream too.
I suspect they would review the things they learned about the space and resources around their nest, making memories of the locations of things more permanent.
This is on the assumption that a purpose of dreaming is to refine and organize memories and things learned during the day.
-
@Gorfram @datarama @alessandro @talya
Ants and bees probably dream too.
I suspect they would review the things they learned about the space and resources around their nest, making memories of the locations of things more permanent.
This is on the assumption that a purpose of dreaming is to refine and organize memories and things learned during the day.
@futurebird @Gorfram @datarama @alessandro @talya it's honestly kind of poetic that humanity keeps having this scientific uncertainty about whether dreaming serves a purpose
-
@Gorfram @datarama @alessandro @talya
Ants and bees probably dream too.
I suspect they would review the things they learned about the space and resources around their nest, making memories of the locations of things more permanent.
This is on the assumption that a purpose of dreaming is to refine and organize memories and things learned during the day.
@Gorfram @datarama @alessandro @talya
We have a common ancestor with arthropods, but it was long long long ago. The fact that dreaming may be shared, even if it's very different by such very different creatures hints at something profound and fundamental about being an organism with a brain and an nervous system and bilateral symmetry who interacts with the world and makes choices.
Having that in common is enough for both of us to need to sleep and also probably need to dream.
