# How can you perform paraphrase mining using Sentence Transformers to find duplicate or semantically similar sentences in a large corpus?

Source: https://zilliz.com/ai-faq/how-can-you-perform-paraphrase-mining-using-sentence-transformers-to-find-duplicate-or-semantically-similar-sentences-in-a-large-corpus
Author: zilliz

To perform paraphrase mining with Sentence Transformers, you start by converting sentences into dense vector representations (embeddings) using a pre-trained model, then compare these embeddings to identify semantically similar pairs. Here's a step-by-step breakdown:

1. **Model Selection and Encoding**: Use a Sentence Transformers model fine-tuned for semantic similarity, such as `paraphrase-MiniLM-L6-v2` or `all-mpnet-base-v2`. These models map sentences to embeddings where similar sentences are closer in vector space. For a corpus of 10,000 sentences, encoding might take minutes on a CPU or seconds on a GPU. Preprocess text by normalizing case and punctuation, but avoid aggressive cleaning to preserve meaning.

2. **Efficient Similarity Search**: Directly comparing every pair of embeddings (10,000² = 100M comparisons) is impractical. Instead, use libraries like FAISS or Annoy to build an index for approximate nearest neighbor search. FAISS's `IndexFlatIP` with cosine similarity allows batch queries—for example, finding the top 5 most similar sentences for each input in milliseconds. Set a similarity threshold (e.g., 0.85) to filter weak matches.

3. **Post-Processing and Validation**: After mining candidate pairs, deduplicate results by keeping only mutual top matches (if sentence A's top match is B, and B's top match is A). For critical applications, add a cross-verification step using models like BERT for natural language inference (NLI) to check entailment. Store results in a lookup table or graph structure for efficient retrieval.

**Example**: For a FAQ system with 50k entries, this pipeline could reduce 50,000² comparisons to 50,000 FAISS queries, identifying clusters like "reset password" and "how to change login" as duplicates. Threshold tuning on a validation set (e.g., 100 manually labeled pairs) ensures precision/recall balance.