Large language models (LLMs), in particular GPTs, have captured the world’s attention over the last few years and led the field of Natural Language Processing (NLP) into its new and improved stage of evolution.
While LLMs may be based on several different neural network architectures, such as transformers and convolutional and recurrent architectures, most of the recent LLMs are based on transformers.
AI and its variants are ultimately an aspirational human attempt to mimic natural intelligence. So it should come as no surprise that the various mechanisms developed so far can be understood and explained in terms of mundane human activities or natural phenomena.
Transformers & Charades
In its simplest form, transformers can be modeled as a game of charades. In a popular party game of gestures and acting, one player is given a word or phrase and has to communicate it through gestures to the team.
If the team can decipher the word or phrase correctly, they win a point.
Transformers consist of an encoder and a decoder.
The player transforming the given word or phrase into a visual representation through basic gestures represents the encoder. The team transforming the gestures into a verbal output represents the decoder.
Transformers use a numerical representation instead of gestures, which is also known as embedding representation.
This embedding captures many different factors of the input information in various dimensions.
Just as the player acting out, say, a longer phrase, decides which words to emphasize to convey the information most accurately and quickly, and may even communicate words out of order for this purpose.
LLMs also weigh the importance of different words or tokens in the input sequence, relative to other words, to facilitate the most accurate capture and communication of context, significance, and long-range dependencies within the sequence.
This is called the self-attention mechanism.
Transformer Architecture
There are different variants of the transformer architecture such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pretrained Transformers).
GPT is designed primarily for generative tasks and focuses only on the decoder portion of the original transformer architecture. This can be thought of as our charades team creating stories based on the input given to them, without a player translating these into gestures.
Meanwhile, BERT specializes in masked word prediction and text classification tasks and utilizes only the encoder portion of the original transformer architecture.
Think of this as the player receiving the input word on a piece of paper and placing this in one box among a set of ten different boxes, based on the characteristics of the input word.
Conclusion
Whether combined or used individually, the encoder and decoder parts of the transformer architecture have immense power to transform the future of not just natural language processing and deep learning, but also humanity in general.
The advent of LLMs, particularly exemplified by GPTs, has marked a significant milestone in the evolution of NLP. By harnessing the power of transformer architectures, these models have not only revolutionized the field but also provided a fascinating glimpse into the intricate workings of human language understanding and generation.
As LLMs continue to advance, they hold the promise of unlocking even greater potentials, bridging the gap between artificial and natural intelligence while reshaping the landscape of linguistic exploration and innovation.