It’s perhaps not obvious that in the example above, the LLM •does• actually do something useful! It conveys information about what’s typical: “When people talk about a goat and a boat and a river, there’s usually a cabbage too. Here are words that typically appear in the ‘answer’ position in such a context.”
What the LLM doesn’t do is actually solve the problem — or even understand the question. Its answer is garbage. Garbage with clues, as in a detective story. But garbage.
I’ve noticed developers often express excitement about LLM assistants when working with unfamiliar tools, and express horror about them when working with tools they know well. That pattern repeats in other domains as well.
It makes sense: “garbage with clues” can be helpful when you’re learning something unfamiliar. It’s truly helpful to hear “When people import [e.g.] Hibernate and say SessionFactory, code like this typically appears next.” That’s useful! Also probably wrong!
Folks could design and market these ML tools around the idea of •identifying patterns• (the thing machine learning is actually good at) instead of •providing answers•. Pure fantasy at this point; too much collective investor mania around the wet dream of the magic answer box. Just noting that a better choice is on the table.
CS / software education / developer training and mentorship needs to redouble its emphasis on •critical reading• of existing code, not just producing code. By critical reading, I mean: “What does this code do? Does it •really• do that? What is its context? What hidden assumptions am I making? How can it break? Does it do what we •want• it to do? We •do• we want it to do? What is our goal? Why? Is that really our goal? What is the context of our goal? How can our larger goal break?” etc.
Add comment