Wrote an essay on the BM25 algorithm that powers Elasticsearch, Apache Solr, and Apache Lucene.
Today, because I had nothing else to do, I went down the rabbit hole of information retrieval, trying to understand why and where TF-IDF eventually hits its limits, how BM25 actually works under the hood, and why systems like Elasticsearch, Solr, and Lucene made it their default ranking function.
Like always, I compiled my learnings into a detailed write-up that covers why BM25 works as well as it does, how each part of the formula influences ranking, and where it fits in a modern retrieval stack.
This write-up should give you a solid mental model to understand BM25 and also IDF of the popular tf-idf; give it a read.