Understanding the Bigram Language Model

The Bigram Mechanism

A bigram language model functions by predicting the next token in a sequence using only the immediate preceding token. The model maintains an embedding table where each row corresponds to a unique token in the vocabulary (65 tokens for the Shakespeare dataset) and each column represents the likelihood of a subsequent token. Because the initial table is randomized, the model produces gibberish until it undergoes training to minimize its prediction error.

Training and Optimization

Training involves processing the dataset in chunks, which are further divided into batches and blocks to allow for parallel computation. The model uses the following process to refine its predictions:

• Softmax Normalization: Raw output values (logits) are converted into probabilities that sum to 1, allowing for an intuitive interpretation of the model's confidence in the next token.
• Negative Log-Likelihood: This loss function measures the distance between the model's predicted probability for the correct token and the target, penalizing the model when it assigns low probability to the actual next token.
• Backpropagation and Optimization: The model uses backpropagation to calculate gradients and an optimizer to adjust the embedding table values. The learning rate must be carefully tuned to avoid unstable updates or excessively slow convergence.

Limitations

While the model can be trained to reduce its loss—often moving from an initial loss of approximately 4.87 to a value near 2.0 after thousands of iterations—it remains fundamentally limited by its architecture. Because it only considers the current token, it lacks the context necessary to form coherent words or long-term structures, necessitating the transition to attention mechanisms used in GPT architectures.