Large Language Models (LLMs) and AI
From Lady Lovelace to Timnit Gebru
The research and interests in language and Artificial Intelligence (AI) are not new. However, advances in computing and storage capabilities ushered in a new era for natural language processing systems. In particular, large language models (LLMs) have recently been in the limelight for performance numbers, startup funding, media hypes, harmful content, social justice and bias issues.
As a result (both due to academic and commercial interests), we are noticing more and more applications of advanced language systems in many diverse settings. Despite critiques, scandals, and market turmoils, investors' interests in funding startups dealing with language models and AI, in general, are not slowing down.
A well researched Forbes story from Mar 27, 2022, lays out the wide scale adoption and interests in language based AI startups from search to writing assistants to healthcare:
We now stand at an exhilarating inflection point. Next-generation language AI is poised to make the leap from academic research to widespread real-world adoption, generating many billions of dollars of value and transforming entire industries in the years ahead.
A nascent ecosystem of startups is at the vanguard of this technology revolution. These companies have begun to apply cutting-edge NLP across sectors with a wide range of different product visions and business models. Given language’s foundational importance throughout society and the economy, few areas of technology will have a more far-reaching impact in the years ahead.
With such positive momentum in the market and investment, this is the wild west. We see companies, after being embroiled in scandals, still operating and receiving fundings. Here is a story of a analytics firm called Loris (that got data from a non-profit partner in crisis hot line):
And the most recent release from OpenAI and Meta turned this LLM business into an arms race:
OPT-175B is the latest entrant in the LLM arms race triggered by OpenAI’s GPT-3, a deep neural network with 175 billion parameters. GPT-3 showed that LLMs can perform many tasks without undergoing extra training and only seeing a few examples (zero- or few-shot learning). Microsoft later integrated GPT-3 into several of its products, showing not only the scientific but also the commercial promises of LLMs.
Of course, not everything here is doom and gloom. From simple text mining to tagging parts of speeches (note that NLP research in U.S academia primary has been English focused), language related AI tools have achieved a lot. These tools could easily now tag and categorise texts, identify patterns, predicting the next word given a specific text (your favourite email clients are using these tools behind the scene). Computers are good at these things and they can do them efficiently.
But is that anywhere close to human intelligence? Why are we even bothering with that? Often, what is called strong AI or Artificial general intelligence (AGI) has also been of interest lately. Some of these ideas come from American philosopher John Searle who published an article in 1980 where he asked “is a machine capable of thinking?” The paper dealt with a thought experiment known as the “the Chinese room”. Basic conclusion is that the syntax is not a sufficient condition for the determination of semantics. But there’s more to it — for a more recent summary and argument check, this article (“Do Computers "Have Syntax, But No Semantics") which complicates this issue even further.
Searle (who, btw, lost his emeritus status for violating the sexual harassment policy of the University of California) is not the first one to question this. Ada Lovelace, a 19th-century mathematician (considered to be the very first computer programmer) noted the limitation of the Analytical Engine:
The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform.
This is known as Lovelace’s objection. From this objection, we now come to daily media and industry hypes that put anything and everything related to AI in the pedestal as the next best thing.
For what? So AI can guess your next word for the email you really do not want to write? Most of the recent LLM models use large dataset but does not guarantee the quality of the datasets. And last couple years lot of research came about outlining that “Bigger isn’t better”.
Besides Ada Lovelace, we should also listen to another smart person: Timnit Gebru (who recently been listed in the 100 most influential people of 2022). Check this elaborate profile about her in Time.
The bottom line is if your input is toxic, the next word guessed by AI will be toxic as well (“Garbage in Garbage out”). Simple probability -- that is how computers work. And because our world and language related mediums are already so problematic, studies have shown even with no toxic content models like GPT-3 answers with toxic elements. AI does not always have to be about social justice or diversity but even on a purely technical aspect some of these LLMs fail. Because we fail to see the connections, and contexts and we fail to understand our complex world, we keep producing LLMs and feel good about human intelligence.
So to conclude, great to have these new achievements with language related AI models. But let’s not try to create human level AIs. There are benefits of large scale models but let’s keep our focus on issues that matter and understand why these models are needed before we spend billions.
Behind these hypes, people like Timnit Gebru, Emily Bender, and Gary Marcus (people that I follow closely) are producing valuable and constructive critiques. Remember, AI is not bad — it is how we use it. And AI is hard (even a few years ago Stanford NLP could not tag “hers” as a pronoun properly) — more memory and more data will not fix the problem. We do not have interdisciplinary collaborations, too much focus on scale and techniques instead of domain expertise.
Technical solutions need to go hand in hand with social, political and ethical understandings. Given the interdisciplinary nature of the applications of artificial intelligence systems, our societal problems cannot be solved by one type of expertise. Computer scientists and engineers should learn about ethics and social science but should not make themselves out to be ethicists or social scientists. Collaborate, learn from each other, and find domain expertise. The issues with LLMs cannot be solved by technical work alone. And listen to smart people like Lady Lovelace and Timnit Gebru.
All the way from Ada Lovelace, and such diverse voices cited. Thank you for taking us on this journey 🪄