Causal vs. Masked Language Models

• GPT language models are causal language models (CLMs), distinct from masked language models (MLMs).
• CLMs predict the next word in a sequence based on preceding words, using an auto-regressive approach.
• MLMs predict a masked word within a sentence based on surrounding context.
• GPT's auto-regressive nature means it sequentially predicts each word based on all prior words in the sequence.
• This allows GPT to generate coherent and contextually relevant text by building upon previous predictions.