Chapter 4: BERT Variants I – ALBERT, RoBERTa, ELECTRA, and SpanBERT

This chapter delves into significant advancements and variations of the BERT architecture, focusing on ALBERT, RoBERTa, ELECTRA, and SpanBERT. We will explore their unique approaches to improving efficiency, performance, and capabilities.

Introduction to ALBERT – A Lite Version of BERT

ALBERT (A Lite BERT) introduces several parameter-reduction techniques to create a more efficient version of BERT without significant performance degradation.

Cross-Layer Parameter Sharing

ALBERT employs cross-layer parameter sharing, meaning that the parameters within each layer are shared across all layers. This dramatically reduces the total number of parameters. Instead of learning unique parameters for each transformer block, ALBERT uses a single set of parameters shared across all blocks.

Factorized Embedding Parameterization

ALBERT decouples the size of the hidden layers from the size of the vocabulary embedding. Typically, BERT uses an embedding size equal to the hidden layer size. ALBERT separates these into two smaller matrices: a wordpiece embedding matrix and a hidden layer embedding matrix. This factorization further reduces parameters, as the embedding size can be kept smaller while still having large hidden layers.

Removing the Next Sentence Prediction (NSP) Task

The original BERT model used the Next Sentence Prediction (NSP) task to learn sentence-level relationships. ALBERT replaces NSP with Sentence Order Prediction (SOP).

Sentence Order Prediction (SOP)

In SOP, the model is given two consecutive segments from the same document and must predict whether they are in their original order or swapped. This task is considered more challenging and more focused on inter-sentence coherence compared to NSP, which often conflated topic prediction with coherence prediction.

Efficient Training Methods

ALBERT's architectural changes, particularly parameter sharing and embedding factorization, contribute to more efficient training by reducing the memory footprint and computational cost.

Training the ALBERT Model

Training ALBERT involves similar procedures to BERT, but with the modifications of SOP and the aforementioned parameter reduction techniques. The efficiency gains allow for training with larger models or on more data within similar resource constraints.

Understanding RoBERTa

RoBERTa (A Robustly Optimized BERT Pretraining Approach) focuses on optimizing the pre-training strategy of BERT to achieve better performance.

Understanding RoBERTa

RoBERTa systematically evaluates and improves upon BERT's pre-training methodology. Key improvements include:

• Dynamic Masking vs. Static Masking: BERT uses static masking, where the masked tokens are chosen only once during data preprocessing. RoBERTa employs dynamic masking, where the masking pattern is generated anew for each input sequence during training. This exposes the model to more diverse masking patterns and improves generalization.

• Training with More Data Points and Large Batch Size: RoBERTa is trained on a significantly larger dataset (160GB of text) and with much larger batch sizes compared to BERT. This scale of training is crucial for its performance gains.

• Removing the Next Sentence Prediction (NSP) Task: Similar to ALBERT, RoBERTa also removes the NSP task, finding that it does not consistently improve downstream task performance and can sometimes be detrimental.

Exploring the RoBERTa Tokenizer

RoBERTa utilizes a Byte-Level Byte Pair Encoding (BPE) tokenizer.

Using Byte-Level Byte Pair Encoding (BPE)

Byte-Level BPE treats the input text as a sequence of bytes, allowing it to handle any text without needing a large vocabulary to cover all possible characters or subwords. This approach is robust to out-of-vocabulary tokens and can effectively represent diverse languages and special characters.

Understanding ELECTRA

ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) introduces a novel pre-training task that significantly improves training efficiency and performance.

Understanding ELECTRA

ELECTRA's core innovation is its "Replaced Token Detection" pre-training task, which is significantly more compute-efficient than masked language modeling.

Generator and Discriminator in ELECTRA

ELECTRA employs a generator-discriminator architecture:

1. Generator: A small masked language model (similar to BERT) that replaces a small percentage of input tokens with plausible alternatives.
2. Discriminator: A masked language model (the main ELECTRA model) that is trained to predict which tokens in the input sequence were replaced by the generator.

Replaced Token Detection Task

The Replaced Token Detection task is performed by the discriminator. For each token in the input, the discriminator outputs a binary classification: whether the token is an original token or a replacement generated by the generator. This task is more sample-efficient than predicting the original masked tokens because the discriminator learns from every token in the input sequence, not just the masked ones.

Training the ELECTRA Model

The training process involves:

1. The generator masks tokens and replaces them.
2. The discriminator processes the corrupted sequence and predicts for each token if it was replaced.
3. The discriminator is trained to minimize the binary cross-entropy loss for this detection task.

ELECTRA's efficiency allows it to achieve state-of-the-art results with significantly less computational resources compared to other BERT variants.

Understanding SpanBERT

SpanBERT is designed to improve the understanding of natural language by focusing on learning continuous spans of text.

Understanding SpanBERT

SpanBERT addresses limitations in BERT's attention mechanism by incorporating span-based objectives during pre-training.

Exploring SpanBERT Applications

SpanBERT has shown strong performance on tasks that require reasoning about contiguous text segments, such as:

• Question Answering
• Coreference Resolution
• Abstractive Summarization

Performing Question-Answering with Pre-Trained SpanBERT

SpanBERT's focus on spans makes it particularly well-suited for question-answering tasks, where identifying the correct answer span within a document is crucial. It achieves this through modified pre-training objectives.

Summary, Questions, and Further Reading

This chapter has introduced key BERT variants: ALBERT for efficiency, RoBERTa for robust optimization, ELECTRA for compute-efficient pre-training, and SpanBERT for span-level understanding. Each variant offers distinct advantages for specific NLP applications and research directions.

Further Reading:

• ALBERT: "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations" (Lan et al., 2019)
• RoBERTa: "RoBERTa: A Robustly Optimized BERT Pretraining Approach" (Liu et al., 2019)
• ELECTRA: "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators" (Clark et al., 2020)
• SpanBERT: "SpanBERT: Improving Pre-training by Representing and Predicting Spans" (Joshi et al., 2019)