Today, I was going through a GitHub blog from 2017, and in it, they shared how they suggest topics for a repository. Here’s how it works…
The pipeline reads the repo name, description, and README, strips out noise like code blocks and file paths, then breaks the remaining text into candidate phrases by removing common stop words.
The stopwords also contain GitHub-specific stop words like “push”, “pull”, and “tool” that appear in almost every repo and carry zero signal.
These candidate words go through a logistic regression classifier trained on ~300 manually labeled examples of “good” and “bad” topics. Phrases like “running slowly” or “performing operations” get filtered out.
The phrases are then scored using tf-idf, where each word’s weight is computed against an IDF dictionary built from all public READMEs. This way, the word “application” scores lower than “assignment” because it shows up far more often across repos. Classic tf-idf.
Finally, near-duplicate topics get collapsed. If “machine learning library” and “machine learning framework” both score high, the lower one gets dropped. This way, the final set is pretty crisp and varied.
This pipeline does not have any heavy NLP or complex parsing - just simple, intuitive functions that just work :) It is actually a pretty breezy read as well.