XXL Language Models and Applications in Data Management (VLDB'22 Tutorial)

The introduction of Transformer-based language models has led to astonishing advances in the domain of natural language processing over the past years. Not only do such models dominate in a variety of standard benchmarks. The latest generation of language models can be specialized to novel, formerly unseen tasks with little to virtually no training data.

In this tutorial, I discuss the two key ideas enabling ultra-large language models: a new neural network architecture, the Transformer, and an unsupervised training process, based on the idea of transfer learning. After discussing the theoretical concepts behind language models, I demonstrate GPT-3 and other models and provide pointers on how to get access to this technology. Finally, I discuss novel use cases in data management that are enabled by language models, covering recent research and open problems.