Positional Embeddings in Transformers: Generalization and Approaches

Transformers, by design, are permutation-invariant, meaning they do not inherently understand the order of tokens within a sequence. To address this, positional embeddings are added to token embeddings to inject information about the sequence's order.

The fundamental concept is: $e_i = x_i + PE(i)$

Where:

• $x_i$: The token embedding for the token at position $i$ (position-independent).
• $PE(i)$: The positional embedding, which encodes the position $i$ into the embedding.

The Challenge of Learned Positional Embeddings and Generalization

When positional embeddings are learned during training, they are typically trained on sequences up to a maximum length, $m_l$. This means the model learns embeddings only for positions $i \le m_l$.

Problem: At inference time, if the model encounters a sequence with a length $m >> m_l$, it lacks learned embeddings for positions $i > m_l$. This deficiency leads to generalization issues, where the model's performance degrades significantly on longer sequences than those seen during training.

Approaches to Generalization

To overcome this limitation, two primary strategies are employed:

1. Extrapolation:

   • Concept: Utilizes a function trained on observed positions to predict values for unobserved positions.
   • Example: If a model was trained on positions 1-10, extrapolation aims to predict meaningful embeddings for positions 11-20 (or beyond) based on the learned pattern.

2. Interpolation:

   • Concept: Maps longer sequences into the range of positions the model was trained on by applying a scaling factor.
   • Example: If a model was trained on positions [1, 10], a sequence of length 20 might have its positions scaled down to the [1, 10] range (e.g., position 15 becomes position 7.5) to fit within the learned embedding space.

Figures illustrating these concepts visually demonstrate extrapolation (predicting beyond the trained range) and interpolation (scaling within the trained range) compared to a baseline with no generalization.

Relative Positional Encoding Methods

Instead of absolute positions, relative positional encoding methods encode the distance between tokens. This is often incorporated directly into the attention mechanism.

The general form of attention with relative positional encoding is: $\alpha(i,j) = \text{Softmax}(q_i \cdot k_j + PE(i,j) + \text{Mask}(i,j)) / \sqrt{d_k}$

Where $PE(i,j)$ is the learned or fixed bias for the offset $i-j$.

T5 Bias (Raffel et al., 2020)

T5 uses learned relative positional biases, grouping offsets into buckets for efficiency and generalization.

• Bucket Strategy:

  • The first $n_b/2$ buckets: Each bucket corresponds to a unique offset (fine-grained detail for short distances).
  • The last $n_b/2$ buckets: Logarithmic bucketing is used for larger distances (coarse-grained detail).
  • A final bucket: Catches offsets beyond a predefined maximum distance, $dist_{max}$.

• Generalization: This parameter sharing across similar offsets allows the model to generalize better, especially when long-range dependencies are rare during training.

ALiBi (Attention with Linear Biases)

ALiBi proposes a non-learned, fixed bias that simplifies positional encoding and enhances generalization.

• Bias Formulation: $PE(i,j) = -\beta \cdot (i - j)$

• Attention Mechanism: The bias is directly added to the attention scores: $\alpha(i,j) = \text{Softmax}(q_i \cdot k_j - \beta \cdot (i - j) + \text{Mask}(i,j)) / \sqrt{d_k}$

• Key Properties:

  • The bias increases linearly with the distance, effectively penalizing attention to tokens further away.
  • No training required for the positional bias term.
  • Generalizes naturally to sequences of any length without needing explicit extrapolation or interpolation mechanisms.
  • The scalar $\beta$ is chosen per attention head, often decreasing geometrically (e.g., $\beta_k = 1 / 2^{8k}$) to prioritize attention to closer tokens in earlier heads.

Rotary Positional Embeddings (RoPE)

RoPE is a more sophisticated method that embeds positional information by applying rotations to token embeddings rather than adding or biasing.

• Embedding Formulation: $e_i = x_i \cdot R(i)$

  Where $R(i)$ is a rotation matrix that depends on the position $i$.

• Mechanism: RoPE models position by rotating vectors in complex space. This method has the advantage of preserving the norm of the embeddings.

• Generalization: RoPE is well-suited for extrapolation because the underlying functions are continuous and not inherently bounded by the training sequence length.

Comparison of Positional Embedding Methods

Note: Some methods like FIRE, Kerple, and Sandwich are variants or extensions with learned or log-scaled functions that aim for generalization.

Conclusion

Modern positional embedding techniques for Transformer architectures prioritize generalization to longer sequences through several key innovations:

• Parameter Sharing: Methods like T5 Bias share parameters across similar positional offsets, improving robustness.
• Avoiding Training: Techniques such as ALiBi and RoPE decouple positional encoding from the training process, allowing them to work effectively on unseen lengths without explicit learning for those lengths.
• Mathematically Scalable Functions: Using inherently scalable functions (like rotations or specific algebraic forms) allows for natural extrapolation beyond training limits.

These advancements are critical for tasks requiring long-context understanding, such as large-scale language modeling and document summarization, where inference sequence lengths frequently exceed those available during training.

---

SEO Keywords

• Positional embeddings in transformers
• Learned vs relative positional encoding
• Rotary positional embeddings (RoPE) explained
• ALiBi positional bias in attention
• Transformer positional encoding generalization
• T5 bias positional encoding buckets
• Relative vs absolute positional embeddings
• Positional encoding for long-context transformers
• Scaling transformers with RoPE and ALiBi
• Efficient positional embedding techniques in NLP

---

Interview Questions

1. Why are positional embeddings necessary in Transformer architectures?
2. What is the limitation of using learned positional embeddings for long sequences?
3. How does relative positional encoding differ from absolute positional encoding?
4. Explain how the T5 model handles positional information using bucketed offsets.
5. What is ALiBi and how does it enable transformers to generalize to longer contexts?
6. How does Rotary Positional Embedding (RoPE) encode position information?
7. Compare RoPE, ALiBi, and T5 positional encodings in terms of generalization and learnability.
8. What are the advantages of using non-learned positional embeddings like ALiBi?
9. How do extrapolation and interpolation help in extending positional embeddings beyond training lengths?
10. Which positional encoding method would you recommend for long-sequence tasks and why?