Mean pooling is often used to create sentence embeddings from transformer token outputs because it provides a simple yet effective way to aggregate contextual information across all tokens. Transformers like BERT generate a sequence of token embeddings, each capturing the context of a specific word within the sentence. However, many downstream tasks (e.g., text classification, semantic similarity) require a single fixed-dimensional vector to represent the entire sentence. Mean pooling averages these token embeddings, combining their information into a single vector that reflects the overall semantic content of the sentence. This approach avoids relying on a single token (e.g., the [CLS] token, which is designed for classification but may not generalize well to other tasks) and instead leverages the full context of all tokens.
The simplicity and computational efficiency of mean pooling make it a practical choice. Unlike more complex methods (e.g., attention-based pooling or learned aggregation layers), mean pooling requires no additional parameters or training. For example, in BERT, the [CLS] token’s embedding is often suboptimal for non-classification tasks unless the model is explicitly fine-tuned for them. By averaging all token embeddings, the method reduces noise from individual tokens while preserving the sentence’s overall meaning. This works well in practice because transformers already encode rich contextual relationships, so even a simple aggregation can retain useful information. For instance, in the sentence “The quick brown fox jumps,” mean pooling would blend the embeddings for “quick,” “brown,” and “fox” into a vector that captures the combined action and subject.
Another advantage of mean pooling is its robustness across diverse tasks. While task-specific pooling strategies might perform better in certain scenarios, mean pooling serves as a strong baseline that generalizes well. For example, in semantic search or clustering, averaging token embeddings often produces embeddings that align well with cosine similarity metrics. This approach also mitigates issues like variable sentence lengths or rare words by distributing their impact across the entire sentence. However, it’s worth noting that mean pooling can dilute the importance of critical keywords—in a sentiment analysis task, words like “terrible” or “amazing” might be underrepresented in the averaged vector. Despite this limitation, its ease of implementation and consistent performance make mean pooling a widely adopted default for generating sentence embeddings in transformers.