# Why are my sentence embeddings coming out as all zeros or identical for different inputs when using a Sentence Transformer model?

Source: https://zilliz.com/ai-faq/why-are-my-sentence-embeddings-coming-out-as-all-zeros-or-identical-for-different-inputs-when-using-a-sentence-transformer-model
Author: zilliz

**Why are my sentence embeddings coming out as all zeros or identical for different inputs when using a Sentence Transformer model?**

**1. Input Preprocessing Issues** 
The most common cause of zero or identical embeddings is incorrect input formatting or preprocessing. Sentence Transformer models require text to be properly tokenized and truncated to the model’s maximum sequence length (e.g., 512 tokens for BERT-based models). If your input text is empty, truncated to zero tokens (e.g., due to aggressive preprocessing like removing all punctuation), or consists of out-of-vocabulary tokens (e.g., rare emojis or URLs), the model may produce zeros. For example, if you pass a list of empty strings or non-text data (like numbers), the encoder will default to zero vectors. Verify that your inputs are valid strings and check for accidental data corruption during preprocessing.

**2. Model Initialization or Loading Errors** 
Identical embeddings for different inputs often occur when the model fails to load correctly or isn’t properly initialized. For instance, if you’re using a custom-trained model, ensure the saved weights are loaded correctly. A misconfigured `mean` or `cls` pooling layer (used to aggregate token embeddings into a sentence embedding) can also collapse outputs to identical values. Test with a pre-trained model like `all-MiniLM-L6-v2` to isolate the issue. If the problem persists, reinstall the `sentence-transformers` library or check for version mismatches with dependencies like PyTorch or HuggingFace Transformers.

**3. Model Collapse During Fine-Tuning** 
If you fine-tuned the model yourself, embeddings may collapse due to training issues. For example, contrastive loss functions like `MultipleNegativesRankingLoss` can cause embeddings to become indistinguishable if the learning rate is too high, the batch size is too small, or there’s insufficient diversity in training pairs. To debug, monitor embedding similarity during training using tools like `cosine_similarity` and add regularization (e.g., `L2` normalization). If embeddings still collapse, try a simpler loss function like `CosineSimilarityLoss` first to validate your pipeline.

**Next Steps** 
Start by testing with a small set of diverse, known-good inputs (e.g., `["Hello world", "This is a test"]`) using a pre-trained model. If embeddings are still zeros or identical, inspect tokenization results with `model.tokenize(text)` to ensure inputs are processed correctly. For custom models, validate the architecture and pooling configuration. If fine-tuning, reduce the learning rate and verify data quality.