Wanted: Advice from CS teachers
-
"Now I'm curious about whether LLMs' code compiles and executes error-free on their first attempt."
At first it did not, but they have added a routine to run it through a compiler until it at least runs without syntax errors and probably produces output that seems like what you asked for for a limited example of input.
This is a bolted on extra check, not some improvement in the base LLM.
But some people are acting like it does represent advances in the LLM.
@futurebird @EricLawton @david_chisnall
"I have improved my LLM"
*looks inside*
*improvement is 100% classical handcrafted algorithmic code* -
@futurebird @david_chisnall In my (very limited) experience, all that resulted in is pseudocode with
//before each lineThey've added some new feature that will pull up a little virtual machine and it will let you run the code in there. It also seems to test that the code will at least compile first.
I worry that people seem to think that the LLM just... "evolved" these features when clearly a human person had to add them.
-
"Now I'm curious about whether LLMs' code compiles and executes error-free on their first attempt."
At first it did not, but they have added a routine to run it through a compiler until it at least runs without syntax errors and probably produces output that seems like what you asked for for a limited example of input.
This is a bolted on extra check, not some improvement in the base LLM.
But some people are acting like it does represent advances in the LLM.
@futurebird Software engineering is the stuff around software coding. The AI-vangelists have decided to re-discover the field from scratch, while expressing increasingly hubristic contempt for the lived experience of the entire field.
-
Wanted: Advice from CS teachers
When teaching a group of students new to coding I've noticed that my students who are normally very good about not calling out during class will shout "it's not working!" the moment their code hits an error and fails to run. They want me to fix it right away. This makes for too many interruptions since I'm easy to nerd snipe in this way.
I think I need to let them know that fixing errors that keep the code from running is literally what I'm trying to teach.
@futurebird a couple things I do. (It sounds like you're already doing some of these with slight variation):
1. Tell them I make errors when programming. My former boss who worked for 30 years as a programmer still makes errors.
Errors are not a bad thing. They are part of programming. Part of the process.
Errors are hints. They are clues to figure out how to make the program work. -
Wanted: Advice from CS teachers
When teaching a group of students new to coding I've noticed that my students who are normally very good about not calling out during class will shout "it's not working!" the moment their code hits an error and fails to run. They want me to fix it right away. This makes for too many interruptions since I'm easy to nerd snipe in this way.
I think I need to let them know that fixing errors that keep the code from running is literally what I'm trying to teach.
@futurebird I tell my students that debugging is 50+% of the work, *even* for me with 10+ years of experience. Then I tell them that what makes me a good programmer is not my ability to write error-free code the first time, but my ability to quickly debug code based on oodles of experience.
You could try intentionally making a few common mistakes and showing off the error messages that result if you're projecting an IDE instead of just writing on the whiteboard.
I've also built a Python library called "optimism" that can do "did you use X construct" or even "X inside Y" which helps write certain kinds of practice problems for beginners. It can generate an error message for "you're not using an if statement inside a loop but the problem requires that" for example. Reading other responses here I'm reminded that I should probably put more effort into the error messages...
-
@futurebird a couple things I do. (It sounds like you're already doing some of these with slight variation):
1. Tell them I make errors when programming. My former boss who worked for 30 years as a programmer still makes errors.
Errors are not a bad thing. They are part of programming. Part of the process.
Errors are hints. They are clues to figure out how to make the program work.@futurebird
2. When a student tells me their code doesn't work, I almost never tell them how to fix it. Here are my steps:
a. Did you read the error? Okay then read the error.
b. What does it say? What line is the error on? Do you know what the words mean? Okay let's break it down. *Go word by word through the error to check their understanding and explain the words they don't know*
c. What part of the line do you think is the problem? How can we change it? What else could we put there instead -
Tangentially related:
"AI can write code so why teach how to code?"
"Great point! It can write an essay too, so why teach how to read."
Like. We've had calculators for decades and still teach arithmetic. And functionally the average person needs to know probably more about mathematics and needs to read more than they did a century ago. The same will apply for code.
@futurebird @david_chisnall As a retired coder. The job description hasn’t changed. Just the ratios on tasks performed. Much more reading code vs lots less writing actual lines of code. I like the sound of this debugging approach, as it forces reading and understanding. I have long said the CS courses needed more reading/reviewing code. Like a large part of traditional art courses teach art “criticism”. You have to be able to tell “good” art from “ bad” art to be able to judge if you work is achieving the goals you want from it. The same applies to code.
-
I think they become anxious when their code isn't working the same as what I have up on the projector and they want to get it fixed RIGHT AWAY so they won't fall behind.
Then when one of them starts calling out they all do it.
I may take some time to explain this.
This never happens when I'm teaching math. Something about coding makes them forget some of their manners, and become less self-sufficient. "It's broke! I'm helpless!"
What is that about?
@futurebird
maybe you already do something like this, but I wonder if some sort of preamble to set expectations around errors will help? Something like "computer programming can look like math but the the instant feedback makes it feel very different. Eventually that will feel like a good thing, but know that you may get error messages when you dont expect them and that a large part of learning to code is learning how to read and understand what those error messages are saying." -
@futurebird Software engineering is the stuff around software coding. The AI-vangelists have decided to re-discover the field from scratch, while expressing increasingly hubristic contempt for the lived experience of the entire field.
Having spent thousands of hours working with various involved parties, trying to understand what they did and how IT could help, I came up with Lawton's Law,
> Everybody's job is a lot more complex than you think it is. Including IT folks' jobs. -
@futurebird
2. When a student tells me their code doesn't work, I almost never tell them how to fix it. Here are my steps:
a. Did you read the error? Okay then read the error.
b. What does it say? What line is the error on? Do you know what the words mean? Okay let's break it down. *Go word by word through the error to check their understanding and explain the words they don't know*
c. What part of the line do you think is the problem? How can we change it? What else could we put there instead3. When I ask if anyone has any questions and a student raises their hand I say "individual?" "Okay I'll help you in a minute"
When they blurt out, I say "let me finish and I'll walk around and help people in a minute". So I never help blurt outs immediately so there's no instant gratification encouraging them to blurt out
-
@futurebird I tell my students that debugging is 50+% of the work, *even* for me with 10+ years of experience. Then I tell them that what makes me a good programmer is not my ability to write error-free code the first time, but my ability to quickly debug code based on oodles of experience.
You could try intentionally making a few common mistakes and showing off the error messages that result if you're projecting an IDE instead of just writing on the whiteboard.
I've also built a Python library called "optimism" that can do "did you use X construct" or even "X inside Y" which helps write certain kinds of practice problems for beginners. It can generate an error message for "you're not using an if statement inside a loop but the problem requires that" for example. Reading other responses here I'm reminded that I should probably put more effort into the error messages...
@futurebird with optimism I can find simpler practice problems for low-level concepts like "write a Boolean expression with two operands that evaluates to True" or "write an assignment statement that combines the result from two function calls" and the checker can enforce those things. I think this kind of low-level practice is sometimes missing because it can't be tested by traditional value-based unit tests.
There's a chicken-and-the-egg problem with debugging: if students don't debug, they'll never learn how to really use/understand a topic. But if they don't understand it, debugging is nearly impossible and the chances of making things worse instead of better are high. Debugging seems easy once you've mastered the relevant concepts, but before that error messages often use unintelligible jargon and add with the "root cause is on another line" problem they can be hard to act on.
About a third of the way through my course, we start to emphasize adding prints to see what wrong with your code. My stock answer to every request for help becomes "Did you add a print to learn more? What did it tell you?" I'd bet if you can get students to do that first because you know that's how you'll respond it would help. A quick "okay add a print and I'll look at your code once In done talking" as a blanket response might work?
-
They are middle school kids totally new to programming so reading the whole thing is a challenge and they will encounter too many words they are still learning.
And really their errors are nearly always typos. Or not putting their function call in main body of the program. Or using a variable they have not defined.
So, for them reading deep isn't that important IMO.
The responses can be very verbose and technical.
I like to compare learning to code to learning a foreign language with a native of that language that doesn’t speak my native language. Similar process with similar challenges.
I would recommend starting with collecting the most common errors, simplifying the problem area, and categorizing them. That should help with pattern recognition, give them a vocabulary to describe their issue beyond just “I’m error,” and provide them with a direction in which to search for a solution.
-
I like to compare learning to code to learning a foreign language with a native of that language that doesn’t speak my native language. Similar process with similar challenges.
I would recommend starting with collecting the most common errors, simplifying the problem area, and categorizing them. That should help with pattern recognition, give them a vocabulary to describe their issue beyond just “I’m error,” and provide them with a direction in which to search for a solution.
And without knowing the group and situation that is the best advice I can provide. After all, Software development is a language puzzle.
-
"Now I'm curious about whether LLMs' code compiles and executes error-free on their first attempt."
At first it did not, but they have added a routine to run it through a compiler until it at least runs without syntax errors and probably produces output that seems like what you asked for for a limited example of input.
This is a bolted on extra check, not some improvement in the base LLM.
But some people are acting like it does represent advances in the LLM.
So I’m a long-time TDD advocate, and I guess this sort of gives up the game? Like, writing trustworthy tests that anticipate every distinct, concrete possibility that matters to us—this has always been The Hard Part of “coding”
-
Now I'm curious about whether LLMs' code compiles and executes error-free on their first attempt.
(Aside from the question of whether it meets requirements, as originally intended; always the hard part).
@EricLawton @futurebird @david_chisnall
There are systems where the LLm will also generate a suite of tests, then repeatedly run the code and the tests, read the error messages and the failed tests, and repeatedly makes modifications. This is slow(can be hours), but it can (for some problems) converge on working code where just asking the chat bot does not.
-
3. When I ask if anyone has any questions and a student raises their hand I say "individual?" "Okay I'll help you in a minute"
When they blurt out, I say "let me finish and I'll walk around and help people in a minute". So I never help blurt outs immediately so there's no instant gratification encouraging them to blurt out
@futurebird
4. As we're coding together, I often type the code wrong on purpose, either because a student suggested the wrong thing, or I take advantage of their ambiguity to make syntactical errors.
And I am careful to not reveal that it's wrong with my tone. So the whole class including me has errors that we work through together. This gets them used to solving errors, shows them that others including me make errors, and they don't expect that the code I show on the screen is always perfect. -
I think they become anxious when their code isn't working the same as what I have up on the projector and they want to get it fixed RIGHT AWAY so they won't fall behind.
Then when one of them starts calling out they all do it.
I may take some time to explain this.
This never happens when I'm teaching math. Something about coding makes them forget some of their manners, and become less self-sufficient. "It's broke! I'm helpless!"
What is that about?
@futurebird To be fair there are a lot of professional software engineers I know who have essentially this same problem. The second anything is wrong they go on slack and start asking for help immediately and go straight to helplessness.
-
@futurebird I gave my students code with errors in it and had them fix it. But they do have to be at a certain level to be able to do that.
I stole this idea from the Java certification exams, which had code examples that would get most people fired.
@venite @futurebird debugging is standard even in primary in many curricula - you can start with basic algorithms, move onto block code, then move onto very basic python and even database/search queries. And still keep the algorithms running alongside!
-
@futurebird maybe reiterate at the start of the demo, and even put up a short statement on the wall so you can point at it rather than answer! Should drum it in quickly
-
The thing I keep saying is: an error message is not a person telling you what specifically went wrong this time. It's a string somebody writing the program months or years ago thought would describe what they *guessed* back then might cause the code to reach that state unexpectedly.
1. Code can always be wrong, sometimes in ways the programmer hadn't thought of (in fact often since they probably handled the ways they'd thought of) and,
2. Error handling code is code.@petealexharris @futurebird @wakame maybe counterintuitive, but I've found that human communication skills are often helpful in learning and understanding programming. Example: "error on line 32" means the interpreter got confused on line 32. Maybe because of line 32 or maybe because there was a miscommunication earlier.
