@futurebird I kinda think the best mental model normies (non-mathematicians/logicians) can easily grasp for LLMs is a cheating partner. For every thing you say, it's going to gaslight you with the most plausible sounding response it can come up with. None of these responses necessarily have anything to do with reality. They're all just chosen based on the likelihood you'll accept them as sounding true.

@marshray @AT1ST

I do agree that some aspects of language might work in that way.

But, when you respond to these posts is that what *you* are doing? Producing the best possible most plausible response for the context?

@AT1ST @futurebird “It’s not actually doing X, it’s just generating text that’s indistinguishable from doing X.”
is classic human cope.

The ability to produce text as it “should look like” is a thing that humans get accredited degrees and good paying jobs for demonstrating.

Good enough is almost always better than perfect, and 80% is usually good enough.

@trochee @futurebird OK, so you just “emitted a word sequence shaped sorta like the word sequences that comes out when metacognition happens.”

And since it’s an argument we’ve all seen before, I can just dismiss your “word sequence” out-of-hand.

See how that works?

@trochee @futurebird OK, so you just “emitted a word sequence shaped sorta like the word sequences that comes out when metacognition happens.”

And since it’s an argument we’ve all seen before, I can just dismiss your “word sequence” out-of-hand.

See how that works?

@marshray @trochee

That's why I didn't ask you how my mind works. I asked you if that was how *your* mind works.

Is it how your mind works?

@futurebird @cxxvii Right. I make this differentiation because A. I want to be clear that no matter how they might advertise this or that method is more accurate, it will always fail due to the underlying issue and B. some people think the tech just isn't fully developed, but its underlying mechanism can NEVER improve without changing to something else entirely.

(Well, as a side note, many do actually make an effort to train in more accuracy, just, the fundamental issue always comes back to bite them. They legit are trying, it just can't work.)

@futurebird @AT1ST The temptation to do that is great.

I try to recognize when I’m posting reflexively and not hit ‘Publish’, because it feels like those posts are largely not adding value.

"Chat GPT told me that it *can't* alter its data set but it did say it could simulate what it would be like if it altered it's data set"

NO. It has no idea if it's telling the truth or not and when it says "I can simulate what this would be like"

This guy is pretty sharp about philosophy but people really really really does not *get* how this works.

"Chat GPT told me this is what it did"

No! It told you what you thought it should say if you asked it what it did!

@CptSuperlative @emilymbender

I have found one use case. Although, I wonder if it's cost effective. Give an LLM a bunch of scientific papers and ask for a summary. It makes a kind of nice summary to help you decide what order to read the papers in.

It's also OK at low stakes language translation.

I also tried to ask it for a vocabulary list for the papers. Some of it was good but it had a lot of serious but subtile and hard to catch errors.

It's kind of like a gaussian blur for text.

@futurebird @CptSuperlative @emilymbender Summaries aren't reliable either.

There are indeed use-cases. But every single one of them comes with caveats. And, I mean, to be fair, most "quick" methods of doing anything come with caveats. It's just that people forget those caveats.

@AT1ST @futurebird “It’s not actually doing X, it’s just generating text that’s indistinguishable from doing X.”
is classic human cope.

The ability to produce text as it “should look like” is a thing that humans get accredited degrees and good paying jobs for demonstrating.

Good enough is almost always better than perfect, and 80% is usually good enough.

@nazokiyoubinbou @CptSuperlative @emilymbender

I don't want to be so dismissive that people who are finding uses for this tech won't pay attention to the important points about the limitations.

People *are* using this tech, some heavily, many probably in ways with pitfalls we won't see the worst results of until it's too late.

Saying "it's just fancy autocomplete" is basically true, but many people think "but autocomplete can't do --"

So, I really try to find ways to "get" this new tech.

@futurebird @CptSuperlative @emilymbender Summaries aren't reliable either.

There are indeed use-cases. But every single one of them comes with caveats. And, I mean, to be fair, most "quick" methods of doing anything come with caveats. It's just that people forget those caveats.

@nazokiyoubinbou @CptSuperlative @emilymbender

If I don't have the experience of "finding it useful" I can't possibly communicate clearly what's *wrong* with asking a LLM "can you simulate what it would be like if you didn't have X in your data set" and just going with the response like it could possibly be what you thought you asked for.

It's not going away.

And right now a lot of people give it more trust and respect than they do other people *because* its a machine.

@CptSuperlative @emilymbender

I have found one use case. Although, I wonder if it's cost effective. Give an LLM a bunch of scientific papers and ask for a summary. It makes a kind of nice summary to help you decide what order to read the papers in.

It's also OK at low stakes language translation.

I also tried to ask it for a vocabulary list for the papers. Some of it was good but it had a lot of serious but subtile and hard to catch errors.

It's kind of like a gaussian blur for text.

@futurebird @CptSuperlative @emilymbender Porting code - small blocks at a time - is the only really useful use-case I've found so far. But at that, it's pretty great: requires a few tries with feedback until "we" get something that compiles and works, but I can port #Python to #Rust that way "easily", so long as I can check each little piece to make sure it compiles and produces the correct output along the way.

@marshray
@futurebird @AT1ST

It's the chinese room problem. Yeah maybe it can generate text that sounds good, but it doesn't know what it's saying in any real sense. And it doesn't know true from false, those are irrelevant concepts to an LLM as it is just using very large statistics to string words together in a way that's statistically plausible.

The machine is fed a shitpost about gluing the cheese on your pizza, now that's an equally valid response for it to give as any real answer bc it has ingested this text.

Chinese room - Wikipedia

(en.wikipedia.org)

@futurebird @CptSuperlative @emilymbender To be clear on this, I'm one of the people actually using it -- though I'll be the first to admit that my uses aren't particularly vital or great. And I've seen a few other truly viable uses. I think my favorite was one where someone set it up to roleplay as the super of their facility so they could come up with arguments against anything the super might try to use to avoid fixing something, lol.

I just feel like it's always important to add that reminder "by the way, you can't 100% trust what it says" for anything where accuracy actually matters (such as summaries) because they work in such a way that people do legitimately forget this.

@neckspike @futurebird @AT1ST In all seriousness, don’t take this the wrong way but:
So what?
Why is that important?
What do you mean by “know what it is saying”?

Do you know what you are saying, or are you just repeating arguments that you have read before?

But maybe you do have a meaningful distinction here.
If so, what question can we ask it, and how should we interpret the response, to tell the difference?

The pizza thing is not particularly interesting because it’s just a cultural literacy test. It’s common for humans new to an unfamiliar culture to be similarly pranked. And that was a particularly cheap AI.