TF-IDF is one of the most widely used ways to quantify document relevance in search engines. While many memorize the formula, the intuition behind why it works is actually pretty simple…
I had written an article about this back in 2020, and it still feels just as relevant and foundational today, in case anyone wants to explore it :) But, here’s the gist
IDF stands for Inverse Document Frequency. The core idea is straightforward: a word that appears in almost every document tells you almost nothing about what makes a specific document relevant.
Words like “a”, “and”, and “the” are everywhere. They carry almost no signal. IDF naturally penalizes them by assigning them a lower weight.
At an intuitive level, the IDF of a term is the log of the inverse probability that a randomly selected document contains that term. (If you feel this line is heavy, I would recommend you read the article; I have explained it in-depth)
So, if a term appears in 10 out of 1,000 documents, its IDF is:
log(1000/10) = log(100)
The rarer the term, the higher its IDF - and the more useful it becomes in distinguishing one document from another. There is also a really neat consequence of looking at this probabilistically.
If you assume terms occur independently, then the IDF of two terms appearing together is simply the sum of their individual IDFs.
That is exactly why search engines can score a multi-word query by adding up the per-term scores. A document containing multiple rare and meaningful terms naturally receives a higher relevance score.
This is such a simple idea, but still one of the most foundational concepts in information retrieval.