One of the most interesting projects that I did in college was to build a search engine from scratch on top of Wikipedia’s data ⚡
The best part was to create the entire search engine ground-up without using Elasticsearch, Solr, or Lucene. I spent weeks building it and going through the nuances of information retrieval.
The dataset was 46GB and parsing, processing, and storing the data on a i3 processor and 4GB of RAM taught me the importance of every single of code we write. It also required me to understand how indexes are stored on the disk and its internal structures.
This is one project I recommend every single one of you to at least attempt. Apart from data management and processing this project will also help you explore ranking algorithms and information retrieval techniques.
By the way, you can find the source code on my GitHub, and remember, this was done back in 2014 during my master’s. So, coding practices are okish.
source code - https://lnkd.in/gkHR8NxX
ps: With the limited processing I had, I kept my laptop near the fan to avoid overheating for the whole night when I built the first index :)
⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
arpitbhayani.me/course