Preface from the new paperback edition of Mathematical Intelligence
The paperback of Mathematical Intelligence has just been published:
https://profilebooks.com/work/mathematical-intelligence/
I penned the preface below some time last year as an attempt to take stock of how the recent surge of LLMs has challenged (and in some cases, reinforced) my thinking on human and machine intelligence.
The hardback edition of Mathematical Intelligence was published in June 2022. It was a simpler time. ChatGPT had not yet tipped generative AI into the mainstream, starting an AI arms race that has been accelerating ever since. Conversational chatbots, limited in their capability, were still the preserve of ardent enthusiasts. Text-to-image tools were only just surfacing, triggering murmurs of discontent among artists who were beginning to find themselves in the crosshairs of automation. And concerns around the existential risk of ‘misaligned AI’ were confined to the margins; they have since taken centre stage, as governments grapple with the question of how we regulate technologies that are developing too rapidly for our institutions to keep pace with.
Interest in AI is at fever pitch. Each week brings new developments, along with countless applications that purport to bolster our productivity and transform our lives. Investment in the space is projected to reach $200 billion by 20251 and is matched only by the buzz surrounding the technology. Humanity finds itself in the grip of a new hype cycle; we are simultaneously excited, terrified and curious at how this chapter in the development of AI will play out – and what it will mean for our species.
The power and versatility of technologies such as ChatGPT caught many people myself included – by surprise. The large language models behind chatbots are, after all, governed by ideas that seem too simple to account for their wide-ranging capabilities. The core function of these models is next token prediction: when you feed a prompt to a chatbot, it selects the word (or ‘token’) that has the highest probability of coming next, based on statistical patterns its model has picked up through the huge amounts of text it has been trained on.
This basic mechanism, applied at the scale of multi-billion parameter models, gives rise to a host of seemingly intelligent behaviours. The paper that accompanied the release of GPT-4, the large language model that powered early versions of ChatGPT, outlined its ability to write poems, generate code, solve complex maths problems, navigate real-world problems and even grasp how people think. Titled ‘Sparks of Artificial Intelligence’, it caused a frenzy among AI commentators who have brought forward their timeframes for when human – and superhuman – levels of machine intelligence will be with us. But while the capabilities of generative AI may have caught us off guard, its major pitfall was fully anticipated: hallucination. The term mistakenly hints at a degree of consciousness within these systems; ‘confabulation’ might be a better description. AI-generated content amounts to an aggregate of messy, unstructured human thought. It is optimised for plausibility rather than truth, and it is rendered in assertive, unquestioning tones. Examples abound of people who have fallen victim to AI’s mistruths, from lawyers who have unwittingly cited false legal precedents generated by chatbots to media outlets that have been forced to retract AI-written articles containing factual errors, to students who have outsourced their homework to these tools without noticing the fabricated citations.
We were warned of the confabulatory behaviours of large language models long before ChatGPT was released. Researchers Emily Bender and Timnit Gebru used the analogy of ‘stochastic parrots’ to capture the mimicry behind these systems’ attempts to replicate the text they have been fed. As long as machines operate solely in the paradigm of pattern-recognition, playing with linguistic structures they do not fully grasp, we remain at the mercy of their concoctions. Generative AI is both incredibly smart and shockingly stupid – it is largely this juxtaposition that makes it so dangerous. Large language models set machines on a ‘mindless’ path towards particular forms of intelligence. Terms like ‘Artificial General Intelligence’ (AGI) are not only poorly defined, they are a distraction, because the present and near-term threats of AI are not predicated on machines acquiring the widest range of cognitive skills. The so-called ‘alignment problem’, for instance – the worry that machines will stray from expected behaviours and pursue goals that might harm humans – exists only because AI cannot be trusted to carry out its objectives sensibly. This does not render machines smarter than humans; only less predictable, more volatile.
No discussion of generative AI is complete without consideration of ‘guardrails’ – the checks and balances that ensure we benefit from the awesome powers of these technologies, without suffering the consequences of their innate stupidity. AI itself is currently unable to address its own blind spots (for instance, language models trained on AI-generated data tend to experience a dramatic deterioration in performance, a phenomenon known as ‘model collapse’). New technological breakthroughs may eventually offer generative AI a way out of its confabulatory rabbit hole, but in the meantime those guardrails must rely on our human expertise. Large language models already depend a great deal on human feedback, both during their initial training and while they are being fine-tuned. While humans are undoubtedly, like AI, capable of both incredible intelligence and shocking stupidity, we can tilt the balance towards the former by deploying our most coveted thinking tools. Mathematical Intelligence is my attempt to illustrate one of the most powerful thinking systems we have at our disposal.
Since the book was first published, however, I’ve been forced to consider whether generative AI has, in fact, taken on the mathematical thinking skills that I explore in the chapters that follow. Generative AI undoubtedly has its moments; the ability to solve Olympiad-level maths problems hints at a capacity for problem solving and reasoning that previous systems lacked. But when the same models that are capable of such impressive feats routinely fail on the most rudimentary problems, or are tripped up by slight changes to the wording of a question, we should be wary of awarding them too much credit. Generative AI flits between the sublime and the ridiculous, but mathematical intelligence is predicated on thought patterns that are more consistently coherent. All mathematicians err, but they do not lay fast and loose with the basic tenets of logic.
It does seem likely that we are on some path towards mathematical automation, and the thought experiments that I close this book with may become reality sooner than I anticipated. But the intelligence race is not being run between machines and humans; presuming human superiority is no more productive than viewing AGI as imminent. Instead, the race will be won by those who figure out how generative AI can augment our ways of thinking and working, rather than undermining or displacing them.
AI has the potential to supercharge our cognitive skills, but it can only do so if we interact with it critically. This means asking meaningful questions of AI tools, employing truth-checking mechanisms to distinguish their meaningful outputs from spurious ones, reining them in when they go off-piste. In this context, the principles of mathematical intelligence are more relevant than ever. And mathematics, as a thinking system, remains one of our best bets for navigating the age of AI.
More on Mathematical Intelligence: What we have that machines don’t:
https://profilebooks.com/work/mathematical-intelligence/