AI-powered
podcast player
Listen to all your favourite podcasts with AI-powered features
Training LLMs and Auto-Regressive Prediction
LLMs are trained by masking words in a text and training a neural net to predict the missing words. The neural net predicts the next word by generating a probability distribution over all possible words in the dictionary and sampling a word based on this distribution. This process is called auto-regressive prediction, which is why these LLMs are referred to as auto-regressive LLMs.