In which I am surprised and confused by claims that large language models can reason, because I cannot see how it could possibly work. By "reason", I mean the mental stunt performed when a philosopher, doctor, lawyer, historian or mathematician draws a conclusion from selected premises, by correct inferences forming an argument.
Not so long ago, I had an entirely unremarkable morning. As usual, I chatted with my wife for a while until she left for work, and then I read some technology news before starting my own work. The neural network people have been busy: their predict-the-next-word models have gotten really good, and now churn out reams of plausible text. "Right on", think I, "good for them", and get back to my coffee largely undisturbed.
Imagine my surprise, then, as over the following months the world collectively loses its shit over these models. Respectable news media print stories about AI with feelings. Government departments start looking worried. Huge quantities of venture capital dollars pour into dubious startups.
All of this does admittedly happen regularly, but several professional computer scientists with bizarre arXiv pre-prints think so too, so it must be true.
Apparently it's all OpenAI's fault: they made their neural network very big, just to see if they could, and now no-one will have a job any more.
The end times are here, and we will be assimilated by the Borg.
At work, I am asked to check if our (largely unrelated) research can be helped by these developments. Friends tell me that the new singularity is not limited to mere random acts of alleged copyright infringement or petty academic misconduct, but it has also developed a sideline in logical arguments. How this happened is not clear: mechanical reasoning is a difficult sine qua non for AGI, and it just...emerged?
By now the corporate-encouraged hysteria has died down a little, although the journalists are still distressed. Most of us have not been replaced or even noticed our new AI overlords, and it looks as if we are on our way down from the hype cycle.
Evidence also begins to pile up that present-day language models cannot in fact do reasoning in the usual sense. No, not if you hook it up to a database. No, not if you give them a scratchpad. No, and I cannot believe I have to say this, not even if you tell them to show their working. Melanie Mitchell's excellent article sums up the state of things if you don't want to read all the breathless press releases. We can all take it easy, the machines will not expel us from Cantor's paradise yet.
I don't want to say that language models are uninteresting or not worth studying, or anything like this. There are plenty of interesting aspects! Language models interact via plain language, which empowers the public to use computation in a completely new way. Their behaviour changes dramatically based on the input text, allowing users to instruct the machine. The problem of giving the machine good instructions produced the properly strange new profession of prompt engineering. You can wire them up with other software to do all sorts of things that you probably shouldn't. Debate continues to rage over the extent of their economic impact. And, of course, they generate human-like text uncannily well.
But they cannot reason.
Probably not in an approximate sense, and certainly not in the sense of checking Hales' proof of the Kepler conjecture, finding the proof of the Robbins conjecture, automation of immigration policy, or even something more mundane like linear programming.
I am certain about this because there is no mechanism by which it could happen. Speak to somebody who believes that it could, and either they know nothing but have a (sometimes literal) vested interest, or they spin you some vague story about representation learning. An argument I have more sympathy with is that since these models read both their input and their output so far, they can work a bit like a Turing machine. The difference is that a Turing machine functions as programmed all of the time, whereas statistical language models are by design slightly incorrect at least some of the time. Over a long enough argument, the probability that a language model reasons fully correctly falls to zero.
In today's terminology, reasoning with LLMs alone doesn't scale.
What I'm still trying to work out is how lots of smart people believed, and in some cases continue to believe, that these stochastic parrots can reason. Perhaps I am completely wrong and the cold, calculating singularity is lurking around the corner after all. In the meantime, stop telling me about "AI reasoning" if you can't also tell me how it does it.