@futurebird Not at face value, no.

But it can actually be pretty good at metacognition, much better than your average human, once it’s pointed in the right direction.

Since current AI’s assist with the training of next generation AI’s, I think there’s a high likelihood of a positive feedback cycle. I don’t think anyone knows where the limit is.

@futurebird The thing everyone should remember is that neural networks are sometimes good at 'doing' the thing they were trained to do, but the GPTs of the world were not trained to produce correct output, they were trained to produce output that is convincing. If they convince you, be wary

I once saw someone who asked their ostensibly locally hosted llm if it was on their machine, and it said no, and they believed it

@cxxvii @futurebird Well, to be clear, the issue isn't the training. A lot of stuff is thrown into the latest training methods to try to make them more accurate. The much more fundamental problem isn't the training, but the actual mechanism itself which -- no matter how good or accurate the training -- simply can't reliably produce the correct output.

@cxxvii @futurebird Well, to be clear, the issue isn't the training. A lot of stuff is thrown into the latest training methods to try to make them more accurate. The much more fundamental problem isn't the training, but the actual mechanism itself which -- no matter how good or accurate the training -- simply can't reliably produce the correct output.

@nazokiyoubinbou @cxxvii

Thank you.

It's like making a machine designed to show people objects that look just like airplanes... and expecting those planes to have engines and be able fly.

But, you never set out to make a program to design machines for flight. Just a program that would show people photos, videos, plans, descriptions that match their *expectations* of airplanes.

@futurebird

I actually started posting again on mastodon (2021-22?) because I was so riled up about aihype and how many people were so deeply fooled about what LLMs are.

@nazokiyoubinbou @cxxvii @futurebird The way that I think about it is "Getting a correct answer doesn't say very much about the likelihood of generating a correct answer in the future."

@futurebird Not at face value, no.

But it can actually be pretty good at metacognition, much better than your average human, once it’s pointed in the right direction.

Since current AI’s assist with the training of next generation AI’s, I think there’s a high likelihood of a positive feedback cycle. I don’t think anyone knows where the limit is.

@futurebird

(For passersby) Some things it seems people have a really hard time grokking are that there are no necessary relations between any training set + prompt combo and a particular result. There is no way to predict what the output will be or to backtrace why the output was what it was.

But, at the same time, the synthetic text is not merely random in relation to the training set + prompt combo. The synthetic text is random within a certain distribution. Humans are terrible at thinking about statistics and aihype plays off of that.

The above does not make LLMs uniquely useful. It makes LLMs uniquely useless. If there's a use-case that LLMs are actually, reliably good at I haven't heard of it.

Today I relaxed by watching this interview with @emilymbender and it's almost painful to watch the hosts's faces as she systematically undercuts just about every use case.

@marshray

When you say "a positive feedback cycle" ...towards what? What is the feedback and what is it going towards positively?

Chat GPT will say things if you ask it what it did, the answers will be similar to texts it has processed about describing how things are done, some in the context of describing how a computer program might do something.

It might even give you a good run down of how LLMs work mixed in there, it might not. But it's not able to ... interrogate it's own process this simply isn't possible. It's not part of how it's designed.

@marshray @futurebird The problem is that it's good at predicting what metacognition should look like, better than your average human who hasn't read the internet's worth of documents on metacognition.

That doesn't mean that it's actually good at metacognition.

@futurebird I think a lot of people are really bad at reasoning about this kind of thing* in general, even before LLMs, and the LLMentalist effect on top of that really sucks them in.

* = multi-layer self-referential language about provenance of language, epistemology, etc.

@AT1ST @futurebird “It’s not actually doing X, it’s just generating text that’s indistinguishable from doing X.”
is classic human cope.

The ability to produce text as it “should look like” is a thing that humans get accredited degrees and good paying jobs for demonstrating.

Good enough is almost always better than perfect, and 80% is usually good enough.

@marshray

And also

> it can actually be pretty good at metacognition, much better than your average human

warrants a giant [citation needed] flag, because the point @futurebird is making is that they are not even doing cognition, let alone metacognition.

At best, they emit word sequences that are shaped sorta like the word sequences that come out when metacognition happens in an actually cogitating thing like a person

@futurebird I kinda think the best mental model normies (non-mathematicians/logicians) can easily grasp for LLMs is a cheating partner. For every thing you say, it's going to gaslight you with the most plausible sounding response it can come up with. None of these responses necessarily have anything to do with reality. They're all just chosen based on the likelihood you'll accept them as sounding true.

@marshray @AT1ST

I do agree that some aspects of language might work in that way.

But, when you respond to these posts is that what *you* are doing? Producing the best possible most plausible response for the context?

@AT1ST @futurebird “It’s not actually doing X, it’s just generating text that’s indistinguishable from doing X.”
is classic human cope.

The ability to produce text as it “should look like” is a thing that humans get accredited degrees and good paying jobs for demonstrating.

Good enough is almost always better than perfect, and 80% is usually good enough.

@trochee @futurebird OK, so you just “emitted a word sequence shaped sorta like the word sequences that comes out when metacognition happens.”

And since it’s an argument we’ve all seen before, I can just dismiss your “word sequence” out-of-hand.

See how that works?

@trochee @futurebird OK, so you just “emitted a word sequence shaped sorta like the word sequences that comes out when metacognition happens.”

And since it’s an argument we’ve all seen before, I can just dismiss your “word sequence” out-of-hand.

See how that works?