If you are deduplicating documents, semantic similarity is probably the wrong tool for the job. I am building a tool to manage my second brain, and there I need to de-duplicate documents (thoughts). Instead of using vector similarity, I am using Jaccard similarity. Here’s why…
Semantic similarity (via embeddings) captures meaning. Two documents can score high similarity even when they share almost no words, because they are talking about the same topic. That is great for search and recommendation, but it is terrible for deduplication.
Jaccard similarity measures the overlap of words or tokens or shingles (short character sequences) between two documents: how many they share versus how many are unique to each. It does not care about meaning at all. It cares about surface-level overlap.
That is exactly what you want when finding near-identical documents. A scraped article and its slightly edited copy will share 80-90% of their 5-grams. Jaccard catches that. An embedding model might not even flag it.
The right mental model: use semantic similarity to find documents that are about the same thing. Use Jaccard to find documents that are the same thing.
Hope this helps.