Token Embeddings in BERT

Before BERT can process textual data, it must first transform raw text into a numerical format. The token embedding layer is a fundamental component of this transformation, responsible for converting individual tokens into dense vector representations. These learned embeddings capture the semantic meaning of each token, forming the initial input layer for BERT.

Understanding Token Embeddings: An Example

Let's illustrate the process with two example sentences:

Sentence A: Paris is a beautiful city. Sentence B: I love Paris.

Step 1: Tokenization

The first step involves breaking down each sentence into smaller units called tokens. Tokenization can split text into words, subwords, or characters, depending on the specific tokenizer used. For simplicity, let's assume the following tokens are generated:

tokens_A = [Paris, is, a, beautiful, city]
tokens_B = [I, love, Paris]

Step 2: Adding Special Tokens

To prepare the input for BERT and enable it to handle multiple sentences, special tokens are added:

• [CLS]: This token is prepended to the beginning of the first sentence. It's a special classification token used for tasks like sentence classification or next sentence prediction, as its final hidden state representation is often used as the aggregate representation of the input sequence.
• [SEP]: This token is appended to the end of each sentence. It serves to demarcate the boundary between sentences, helping BERT distinguish between them, especially in tasks involving sentence pairs.

Applying these special tokens, our token lists become:

tokens_A_processed = [[CLS], Paris, is, a, beautiful, city, [SEP]]
tokens_B_processed = [[CLS], I, love, Paris, [SEP]]

When processing multiple sentences, BERT concatenates them with a [SEP] token in between and a [CLS] token at the very beginning:

final_tokens = [[CLS], Paris, is, a, beautiful, city, [SEP], I, love, Paris, [SEP]]

Step 3: Token Embedding Layer

Each token in the final_tokens list is then passed through the token embedding layer. This layer maps each unique token to a fixed-size, dense vector. These vectors are learned during the BERT model's training process, meaning the model adjusts their values to best represent the tokens in a way that aids in downstream tasks.

For instance:

• E[CLS] represents the embedding vector for the [CLS] token.
• E[Paris] represents the embedding vector for the token "Paris".
• And so on for all other tokens.

Crucially, all token embeddings generated by this layer have the same dimensionality (e.g., 768 for BERT-base). This consistent dimensionality is essential for the subsequent layers of the BERT model, which operate on fixed-size inputs.

Summary

Token embeddings constitute the initial layer of BERT's input representation. They are responsible for converting discrete textual tokens into continuous, dense vector representations that capture the identity and some semantic meaning of each token. These token embeddings serve as the foundational input upon which other embedding types, such as segment embeddings (to distinguish between sentences) and position embeddings (to indicate token order), are added to create the complete input representation for the BERT model.

SEO Keywords

• Token embeddings in BERT explained
• How BERT converts text into vectors
• Role of [CLS] and [SEP] tokens in BERT
• BERT tokenization and embedding process
• Word to vector transformation in BERT
• Dense vector representation in NLP
• BERT input preparation example
• Embedding layer in transformer models

Interview Questions

• What is a token embedding in the BERT model?
• How does BERT use tokenization before embedding input?
• What are the purposes of the [CLS] and [SEP] tokens in BERT?
• Why do all embeddings in BERT have the same dimensionality?
• Describe the process of generating token embeddings for a sentence in BERT.
• How are subword tokens handled in BERT’s embedding layer?
• What is the embedding dimension in BERT-Base?
• How does BERT learn token embeddings during training?
• Can two identical tokens (like “Paris”) have different embeddings in BERT? Why?
• How do token embeddings contribute to BERT’s contextual understanding?