BERT Pre-training Procedure

BERT (Bidirectional Encoder Representations from Transformers) has significantly advanced Natural Language Processing (NLP) through its innovative pre-training strategy. This documentation explores BERT's pre-training methodology, the datasets utilized, and the intricate procedures that enable it to learn powerful contextual language representations.

Datasets Used for BERT Pre-training

BERT is pre-trained on two extensive and diverse text corpora:

• Toronto BookCorpus: A collection of books providing a wide range of narrative and descriptive language.
• English Wikipedia: A vast repository of encyclopedic knowledge, covering numerous topics and writing styles.

These datasets offer a rich variety of linguistic structures, vocabulary, and semantic nuances, which are crucial for BERT to learn generalized language patterns.

Pre-training Objectives: MLM and NSP

BERT's pre-training relies on two self-supervised learning tasks:

1. Masked Language Modeling (MLM)

This task is also known as the "cloze task." In MLM, a certain percentage of tokens in a given input sequence are randomly masked. The model is then trained to predict the original identity of these masked tokens based on their surrounding context. This forces BERT to understand the relationships between words in both directions (left and right context).

2. Next Sentence Prediction (NSP)

NSP is a binary classification task where BERT learns to determine if a second sentence, labeled B, logically follows a first sentence, labeled A. This objective helps BERT understand relationships between sentences, which is vital for tasks like question answering and natural language inference.

Sampling Sentences for Pre-training (NSP Data Preparation)

To prepare data for the NSP task, sentences are sampled from the corpus according to the following procedure:

1. Sentence Pair Sampling: Two text spans (sentences A and B) are sampled from the corpus.
2. Token Count Constraint: The total number of tokens from both sentences (A and B) must not exceed 512.
3. Labeling:
   • With 50% probability, sentence B is a true successor to sentence A. This pair is labeled as isNext.
   • With the remaining 50% probability, sentence B is a randomly selected sentence from the corpus. This pair is labeled as notNext.

Example:

• Sentence A: "We enjoyed the game."
• Sentence B: "Turn the radio on."

In this example, Sentence B is not a continuation of Sentence A. Therefore, this pair would be labeled as notNext.

Tokenization and Special Tokens

Before being fed into the BERT model, the sampled text sequences undergo tokenization and the addition of special tokens:

1. WordPiece Tokenization: BERT utilizes a WordPiece tokenizer, which breaks down words into subword units. This helps handle out-of-vocabulary words and reduces the overall vocabulary size.
2. Special Token Insertion:
   • [CLS] Token: A special classification token is prepended to every input sequence. The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks.
   • [SEP] Token: A separator token is appended at the end of each sentence (or segment). It marks the boundary between two segments, particularly important for the NSP task.

Example Token Sequence:

For the pair "We enjoyed the game." and "Turn the radio on.", the tokenized input sequence with special tokens would look like:

[CLS], we, enjoyed, the, game, [SEP], turn, the, radio, on, [SEP]

Applying Masked Language Modeling (MLM)

During pre-training, BERT applies the MLM objective by masking a percentage of the tokens in the input sequence. A specific strategy, often referred to as the "80-10-10 rule," is employed for replacing the selected tokens:

• 80% of the time: The chosen token is replaced with the [MASK] token.
• 10% of the time: The chosen token is replaced with a random token from the vocabulary.
• 10% of the time: The chosen token remains unchanged (kept as the original token).

This strategy prevents the model from solely relying on the [MASK] token and encourages it to learn robust representations even when input tokens are not perfectly masked.

Example (masking "game"):

[CLS], we, enjoyed, the, [MASK], [SEP], turn, the, radio, on, [SEP]

BERT is then trained to predict the original token for [MASK] and simultaneously perform the NSP task to classify the relationship between sentence A and sentence B.

Training Configuration and Hyperparameters

BERT's pre-training is a computationally intensive process. A typical configuration includes:

• Batch Size: 256 sequences per batch.
• Training Steps: Approximately 1 million steps.
• Optimizer: Adam optimizer is commonly used.
• Learning Rate: An initial learning rate of 1e-4 (e.g., 0.0001).
• Beta Values for Adam: Typically set to (β₁, β₂) = (0.9, 0.999).
• Warm-up Steps: 10,000 iterations.

Learning Rate Warm-up

A crucial aspect of stable training is the use of a learning rate warm-up strategy:

• Purpose: In the initial stages of training, a very high learning rate can lead to instability and divergence. Warm-up gradually increases the learning rate from a very small value (or zero) to the maximum target learning rate over a specified number of steps.
• Mechanism: The learning rate increases linearly during the warm-up period.
• Decay: After the warm-up phase, the learning rate typically decays linearly to facilitate fine-tuning and convergence.

This technique, known as learning rate scheduling, ensures that the model trains efficiently and stably.

Regularization Techniques

To prevent overfitting and enhance generalization capabilities, BERT employs several regularization techniques:

• Dropout: Applied to all layers with a probability of 0.1. Dropout randomly deactivates a fraction of neurons during training, forcing the network to learn more robust and distributed representations.
• Activation Function: BERT uses the GELU (Gaussian Error Linear Unit) activation function. GELU combines desirable properties of both ReLU and sigmoid functions, offering a smoother, non-linear activation.

GELU Function

The GELU activation function is defined as:

$$ \text{GELU}(x) = x \cdot \Phi(x) $$

Where $\Phi(x)$ is the cumulative distribution function (CDF) of the standard normal distribution.

A commonly used approximation for GELU is:

$$ \text{GELU}(x) \approx 0.5 \cdot x \cdot \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} \cdot \left(x + 0.044715 \cdot x^3\right)\right)\right) $$

This smooth non-linearity aids BERT in capturing complex patterns and dependencies within language more effectively than simpler activation functions like ReLU.

Conclusion

The pre-training procedure of BERT is a highly effective strategy that combines Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) on massive text datasets. Through meticulous data sampling, tokenization, strategic masking, carefully chosen hyperparameters, learning rate scheduling, and robust regularization techniques like dropout and GELU activation, BERT is empowered to learn powerful, contextualized language representations. This foundational pre-training makes BERT a versatile model capable of achieving state-of-the-art performance on a wide array of downstream NLP tasks, including sentiment analysis, question answering, named entity recognition, and many more.

---

SEO Keywords for BERT Pre-training

• BERT pre-training process
• BERT training datasets
• Masked Language Modeling in BERT
• Next Sentence Prediction BERT
• BERT tokenizer WordPiece
• BERT training hyperparameters
• Learning rate warm-up in BERT
• BERT regularization techniques

Interview Questions on BERT Pre-training

• What datasets are commonly used for pre-training BERT?
• Explain the two main pre-training objectives of BERT.
• How does the Next Sentence Prediction (NSP) task help BERT learn sentence relationships?
• What is the role of the [CLS] and [SEP] tokens in BERT input?
• How does the Masked Language Modeling (MLM) task work in BERT pre-training?
• What is the 80-10-10 masking strategy in BERT?
• Why is learning rate warm-up important during BERT training?
• What optimizer and hyperparameters are typically used to train BERT?
• How does the GELU activation function differ from ReLU, and why is it used in BERT?
• What regularization techniques does BERT use to avoid overfitting during training?