Behind the scenes of ChatGPT's unrivalled notoriety

Have you wondered how ChatGPT came to be so popular? What triggered such sudden notoriety? Well, we have! With that question in mind, we came down to the Criteo AI Lab (CAIL) looking for answers and had the chance to meet with Liva Ralaivola, Head of the AI Innovation & Research group within CAIL. There, we talked about Generative AI and learned, among other things, that Large Language Models (LLM) that power the likes of ChatGPT are not new to the international science community. They have been around for a few years already. Yet, ChatGPT is on every lip. So, what is the fuss all about? Here is what we learned from Liva!  

What is AIR?  

AIR stands for AI Innovation & Research, a group that I have the honor to lead. It is a fantastic group of 50+ highly AI-devoted engineers and researchers within the Criteo AI Lab (CAIL). We use our expertise and skill sets to monitor, research, and advance the science and technology of AI and see how to transfer the latest AI developments to prod. In a nutshell, our mission is to identify and test the AI models and algorithms with the best potential to bring our product offering to the next level. Our challenge, as an AI-heavy tech company, is to stay alert to keep up with the breakneck pace of the industry and the science behind it and to be aware of the latest trends and needs of the market, as much as anticipate those that might arise.  

At CAIL – and thus in AIR –, we have created a win-win-win situation, where engineers and researchers feed each other from their experience of deploying AI at scale and pushing the state of the art of AI, respectively, the ultimate winner being Criteo.   

A few pointers about Generative AI and Large Language Models  

Down at the lab, we all share a common passion and purpose: Artificial Intelligence (AI) or, more precisely, the subfield of AI that is called Machine Learning (ML) -- if not scientifically correct, the two terms are today used in an almost interchangeable way. We are fond of the tech and science at the core of this field. AI is the birth territory of ChatGPT. More precisely, it is the mother country of what is known as Large Language Models (LLM), a family of mathematical models on which ChatGPT and other advanced chatbots hinge. In a sense, if today we have the opportunity to interact with chatbots that are a lot more sophisticated than those that have been around for decades, it is because they are equipped with an LLM “brain” that processes the bits of information contained in a text (e.g. conversation, document to summarize, word to be defined) and produces articulate answers. ChatGPT provides an interface between humans and an under-the-hood LLM. And you know what? Even if ChatGPT gained crazy attention at the beginning of 2023 and led us to write a whitepaper on it, the fact is that the scientific community has been studying Large Language Models for a few years now. Indeed, LLMs like Generative Pre-trained Transformers models (or GPT, as the second part of the name ChatGPT) have been investigated since 2017, when a research team from Google introduced them for the first time at the well-known NeurIPS scientific conference. And the AI community has developed a tremendous amount of LLMs since then. The curious reader may bear in mind that LLMs are ML models and, as such, they improve their capabilities (to predict the next word in a sequence of words) when fed with training data, textual data here; the main prowess of ChatGPT’s LLM is for it to have been trained on a quantity and variety of text data never tackled before.   

ChatGPT: A paramount innovation?  

As said, ChatGPT provides a way to take advantage of and extoll the power of LLMs. Almost all the data publicly available on the web were used to train the LLM over which ChatGPT was built, making this chatbot sort of a multilingual universal scholar, capable of answering almost any question with a level of language proficiency that no automated system has ever attained before. ChatGPT is a paramount innovation: it assembles existing pieces of technology (e.g. chatbot systems) and nuggets of science (LLMs) to provide an easy-to-use tool/service that was beyond the reach of our imagination before.   With Hocine Slimani, project manager in CAIL and Patrick Gallinari, researcher in AIR and professor at Sorbonne University, we dug deeper into the ChatGPT topic to raise awareness among our Criteo community about the wealth of opportunities and challenges opened by ChatGPT-like apps. A critical topic to understand that we pinpoint was how the advent of ChatGPT would impact search engines and the monetization of search, as well as the (new) way users would interact with the new generation of AI-based search systems. You can get a sense of what this topic relates to by experiencing with Bing search engine: it integrates the possibility to make conversational search, and it does not only issue a set of URLs in response to a query, but instead produces Natural Language-like answers containing pointers to what the user is searching for.  

From the Criteo perspective, rethinking the monetization model of search pertains to the potential enhancement of our Retail Media/C-Max products (among others), which precisely rely on search/recommendation; profits to be gained from a conversational search integration are to be investigated and sized. Beyond search, the past year has witnessed the deployment of copilot/AI assistants in spreadsheet apps, word processors, and even coding environments. As envisioned in our article about ChatGPT, the number of applications of ChatGPT-like engines is endless, and tech people have already started to deploy them at scale. From a broader point of view, it is all the potential benefits of Generative AI (GenAI) that we need to understand, knowing that LLMs are just one family of GenAI models, just by the side of other models built to design prompt-guided images, sounds, and videos. So it reinforces the need for us, as a Tech company, to work on identifying all the opportunities that AI assistance brings: as mentioned, this could pertain to our product offer (e.g. the backend search/recommendation and bidding engine), to new features that we could propose to our clients (e.g. AI-assisted visual assets creation), and even to the optimization of our service delivery and our operational excellence (e.g. around the clock customer support, automated leads identification). Beyond the sheer opportunities related to what we can build, there is an expectation from the ecosystem to see where we stand, as a company, regarding our ability to make the best of the latest AI developments.   

A groundbreaking yet challenging tool  

ChatGPT has initiated a profound paradigm shift on a social and business level. Riding this wave, we face the challenge to identify a “killer” use case, that i) would differentiate us from the competition, ii) answer/spur clients' needs, iii) leverages the wealth of data we collect on a daily basis iv) can scale on our platform and hardware, and v) complies with the latest regulations on data protection (cf. intellectual property, right to be forgotten) and ethical principles. Such an innovative breakthrough will only be possible if it results from a cross-department effort, gathering diverse profiles from the commercial, Product, and R&D teams.  

However great, ChatGPT and other GenAI-based tools bring their load of challenges. The challenge of creativity and of finding the best-unimagined use. The challenge of deployment of sophisticated AI models, from an energy-consumption point of view. The challenge of ethical concerns is, for those models, not to reinforce biases in the training data. The challenge of controlled hallucination is, for those models, not to create apparently plausible yet false text/images/videos. To me, these challenges are not a "no-go", but on the contrary, they have made AI even more appealing than before, posing exciting questions, and leading to wonder what the future of Generative AI will look like. 

A column by Liva Ralaivola, Head of the AI Innovation & Research group within CAIL.