"Chat GPT told me that it *can't* alter its data set but it did say it could simulate what it would be like if it altered it's data set"
-
@futurebird @CptSuperlative @emilymbender Summarisation is the one I’d be most nervous about because creating a summary is hard: it requires understanding the content and knowing which parts are relevant in a given context, which is why LLMs tend to be awful at it. They don’t understand the content, which is how you get news summaries that get the subject and object the wrong way around in a murder. They don’t know what is important, which is how you get email summaries that contain a scam message and strip all of the markers that would make it obvious that the message is a scam.
If you’re going to read all of them and are just picking an order, that’s probably fine. The worst that a bad summary can do is make you read them in the wrong order and that’s not really a problem.
@david_chisnall @futurebird @CptSuperlative @emilymbender btw, we are working on "edge AI" in a research project and have taken to reframe "summarization" of the source information into "wayfinding" of the information.
Offering fidelity estimations and affordances to navigate the source text from the condensed version should (we hope) inform the mental model of users that they are using a stochastic machine that is merely there to help work with large texts. Still early working hypothesis.
-
@david_chisnall @CptSuperlative @emilymbender
It can summarize scientific papers well in part because they have a clear style and even come with an abstract.
The words and phrases in the abstract of a paper reliably predict the content and main ideas of the paper.
Moreover, even if you remove the abstracts, it has lots of training data of papers with abstracts.
@futurebird @david_chisnall @CptSuperlative @emilymbender
But the abstract is already a summary of the paper you can scan to tell if the paper will be useful to you, and you can (usually) trust that summary to be accurate to the content of the paper and concise enough to include the most relevant points. You can't assume the same of an LLM summary, so it's worse than an abstract search.
I can see the advantage of a syntax- and context-aware abstract search if LLMs were that, but they aren't.
-
@Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender
If I asked chat GPT to "turn off the porch light" and it said "OK, I've turned off the light on your porch." I would know that it has not really done this. It has no way to access my porch light. I would realize that it is just giving a text answer that fits the context of the previous prompts.
So, why do people think it makes sense to ask chat GPT to explain how it produced a response?
@futurebird @Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender because they secretely hope it will turn off the porch light and then will do your bidding and take over the world for you.
-
@Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender
If I asked chat GPT to "turn off the porch light" and it said "OK, I've turned off the light on your porch." I would know that it has not really done this. It has no way to access my porch light. I would realize that it is just giving a text answer that fits the context of the previous prompts.
So, why do people think it makes sense to ask chat GPT to explain how it produced a response?
@futurebird @Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender sigh the stochastic parrots paper (thanks Emily! <3) did an excellent job of explaining the reason. astonishingly, it did so before this was a widespread phenomenon.
-
@futurebird @Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender sigh the stochastic parrots paper (thanks Emily! <3) did an excellent job of explaining the reason. astonishingly, it did so before this was a widespread phenomenon.
@futurebird @Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender
people assess credibility by building a mental model of the person they're talking to, based on how they speak and what they say. the machine subverts that process by being something we don't have models for.
-
@Jirikiha @nazokiyoubinbou @joby @CptSuperlative @emilymbender
If I asked chat GPT to "turn off the porch light" and it said "OK, I've turned off the light on your porch." I would know that it has not really done this. It has no way to access my porch light. I would realize that it is just giving a text answer that fits the context of the previous prompts.
So, why do people think it makes sense to ask chat GPT to explain how it produced a response?
@futurebird Remember when it was going around that ChatGPT couldn't count the number of letters in a given word? Like, saying Raspberry had 2 R's?
It's because it breaks words down into chunks, not letters, for some unfathomable reason. Thing is, if you asked it how it figured that out, it would demonstrate that it broke the word down into individual letters, then count each letter, and then get a different answer that might ALSO still be wrong, somehow, and then go "See? Like that."
-
@futurebird @david_chisnall @CptSuperlative @emilymbender
But the abstract is already a summary of the paper you can scan to tell if the paper will be useful to you, and you can (usually) trust that summary to be accurate to the content of the paper and concise enough to include the most relevant points. You can't assume the same of an LLM summary, so it's worse than an abstract search.
I can see the advantage of a syntax- and context-aware abstract search if LLMs were that, but they aren't.
@petealexharris @david_chisnall @CptSuperlative @emilymbender
I want you to give it a try. Take one of those folders of pdfs of papers you are "gonna totally read" and give them to https://notebooklm.google.com/
Ask for a summary. You are correct about the limitations and that's better IMO than not understanding them, but the quality of these guesses is very good and useful in the right contexts. Until I saw this I couldn't understand why so many people were using it at all.
-
@futurebird Remember when it was going around that ChatGPT couldn't count the number of letters in a given word? Like, saying Raspberry had 2 R's?
It's because it breaks words down into chunks, not letters, for some unfathomable reason. Thing is, if you asked it how it figured that out, it would demonstrate that it broke the word down into individual letters, then count each letter, and then get a different answer that might ALSO still be wrong, somehow, and then go "See? Like that."
Because when you ask it "how did you come up with that answer" it looks at the vast data set for examples of people explaining how they come up with answers and then it produces an answer similar to what people have said.
Maybe part of what trips people up is the hubris of thinking you can ask the LLM a question it has no training data for. The training data is so huge that this is really unlikely.
-
@petealexharris @david_chisnall @CptSuperlative @emilymbender
I want you to give it a try. Take one of those folders of pdfs of papers you are "gonna totally read" and give them to https://notebooklm.google.com/
Ask for a summary. You are correct about the limitations and that's better IMO than not understanding them, but the quality of these guesses is very good and useful in the right contexts. Until I saw this I couldn't understand why so many people were using it at all.
@petealexharris @david_chisnall @CptSuperlative @emilymbender
Is this an efficient use of electricity and computing power? This is a good question and the answer may be "no."
-
Yes, exactly.
They seem (willingly? accidentally?) to think that they are having an earnest two-way conversation, when they're really just watching a non-sentient spreadsheet change numbers depending on what they say.
@FediThing @futurebird the metaphor I've used with my "non-technical" friends and family is that LLMs are basically like billions of Galton Boards that each have one hole for a marble on the input side and one output hole for every word and punctuation mark in every language.
Connected to each of the output holes is another Galton Board with it's input.While it's a gross oversimplification that's ignoring context and attention, and is really better suited to explain a Markov Chain, so far it has helped me drive home the point that it's "just" stochastically picking the "most correct" word given a preceding word.
It's also useful to visualize why training is so expensive: You have to tune every peg of every Galton Board until the system behaves correctly.
-
"Chat GPT told me that it *can't* alter its data set but it did say it could simulate what it would be like if it altered it's data set"
NO. It has no idea if it's telling the truth or not and when it says "I can simulate what this would be like"
This guy is pretty sharp about philosophy but people really really really does not *get* how this works.
"Chat GPT told me this is what it did"
No! It told you what you thought it should say if you asked it what it did!
@futurebird There's always been a small minority of people who get overly taken in with say, Eliza and other ancient chatbots whose nonsense isn't anywhere nearly as plausible.
Now that the language generation is much more consistently plausible, I guess in retrospect it doesn't surprise me that much that so many more people would get taken in so easliy.
-
@futurebird I wish I could.
But it would just make people angry, and I feel like I’ve done enough of that today already.I'm not angry, I'd just really like to know what you are getting at.
-
@cykonot @futurebird There was a really neat podcast called the Sci Phi podcast where a grad student was interviewing a bunch of philosophers of science and I loved it, but I can't find it any more
Edit: looks like it is still going! https://sciphipodcast.org/podcast
@semitones @cykonot @futurebird Robinson's Podcast has a lot of good philosophy of science episodes, with a particular focus on foundations of physics but per OP there was a recent episode w/ Ned Block that is one of many that touches on LLMs and philosophy of mind: https://youtu.be/wM1fcZr0iSk
