Whole Word Masking (WWM) in BERT

When training BERT using the Masked Language Modeling (MLM) task, a common and effective variation is Whole Word Masking (WWM). This technique aims to improve the model's ability to understand full word semantics by ensuring that entire words are masked if any part of them is selected for masking. This contrasts with standard token masking, which might only mask a subword unit.

How Whole Word Masking Works

Let's break down the process with an example.

Example: Masking the Sentence "Let us start pretraining the model"

Step 1: Tokenization Using WordPiece

BERT employs the WordPiece tokenizer, which breaks down complex or rare words into smaller subword units. After tokenization, our example sentence might look like this:

tokens = [let, us, start, pre, ##train, ##ing, the, model]

Step 2: Adding Special Tokens

For proper input formatting in BERT, special tokens are added:

tokens = [ [CLS], let, us, start, pre, ##train, ##ing, the, model, [SEP] ]

• [CLS]: Marks the beginning of the sentence.
• [SEP]: Denotes the end of the sentence.

Step 3: Masking 15% of Tokens

In a typical MLM task, approximately 15% of the tokens are randomly selected for prediction. Suppose that during this random selection, the subword ##train is chosen.

With Whole Word Masking, this selection triggers the masking of the entire word that ##train belongs to – in this case, "pretraining". This means pre, ##train, and ##ing are all masked.

The updated token list becomes:

tokens = [ [CLS], let, us, start, [MASK], [MASK], [MASK], the, model, [SEP] ]

Key Insight: If any subword unit of a word is selected for masking, the entire original word is masked. This encourages the model to learn the meaning of complete words rather than just isolated subword patterns.

Step 4: Adjusting the 15% Masking Rate

It's crucial to maintain the overall 15% masking rate across the entire sequence. Masking entire words might sometimes cause this rate to be exceeded. To address this, the WWM strategy may skip masking other selected words to keep the overall percentage consistent.

For instance, if masking "pretraining" already brings the total masked tokens close to or at the 15% threshold, the model might choose not to mask another word like "let" that might have also been randomly selected.

The final tokens, after adjusting for the rate, would be:

tokens = [ [CLS], let, us, start, [MASK], [MASK], [MASK], the, model, [SEP] ]

(Note: In this specific example, the adjustment step might not change the output if the initial selection naturally results in around 15% of the tokens being masked, but it highlights the adjustment mechanism).

Why Use Whole Word Masking?

Whole Word Masking offers significant advantages for BERT's pretraining:

• Improved Word Semantics: WWM helps the BERT model learn more coherent and accurate representations of complete words, especially for compound words or morphologically rich terms (e.g., "pretraining", "unhappiness").
• Enhanced Downstream Task Performance: By understanding full words better, WWM often leads to improved performance on various downstream Natural Language Processing (NLP) tasks such as:
  • Sentiment Analysis
  • Question Answering
  • Named Entity Recognition
  • Text Classification

Summary

• Core Principle: Whole Word Masking (WWM) ensures that if any subword unit of a word is selected for masking, the entire original word is masked.
• Benefit: This method significantly improves the model's ability to understand full-word context and semantics compared to standard subword masking.
• Rate Adherence: WWM strategies are designed to adhere to the overall 15% masking rule by making intelligent adjustments to token selections.
• Outcome: WWM makes BERT more adept at predicting complete words, leading to stronger linguistic comprehension and better performance on real-world NLP applications.

SEO Keywords for Whole Word Masking (WWM) in BERT

• Whole Word Masking BERT
• BERT masking strategies
• WordPiece tokenizer masking
• BERT pretraining techniques
• Improving BERT with WWM
• Masked Language Modeling enhancements
• Subword vs whole word masking
• BERT NLP task performance

Interview Questions on Whole Word Masking (WWM)

1. What is Whole Word Masking (WWM) in BERT, and how does it differ from standard token masking?
2. How does the WordPiece tokenizer affect the masking process in BERT, particularly with WWM?
3. Why does WWM mask entire words instead of individual subwords?
4. Can you explain how WWM maintains the 15% overall masking rate during training?
5. How does Whole Word Masking improve BERT’s understanding of word semantics?
6. What impact does WWM have on downstream NLP tasks like question answering or sentiment analysis?
7. Describe the tokenization and masking steps involved in Whole Word Masking.
8. Why might standard subword masking be less effective than WWM in certain contexts?
9. How does masking the entire word help the model capture morphological information?
10. Are there any trade-offs or challenges associated with using Whole Word Masking during BERT pretraining?