I extensively used Elasticsearch for 2 years straight while I was leading and building Search, here’s one of the practices that helped us keep the service efficient and performant ⚡
Each document (for a lesson/video) in the ES cluster had a title, description, and total_views attributes. While name and description are leveraged for full-text search, “total_views” is used to boost the document to rank higher.
The attribute “total_views” is incremented every time someone watches/plays the content, and this is where the problem becomes interesting. For really popular content, we might see more than a million updates per minute kicking in for the document. There are two ways to update the document: full update and partial update.
In Full Update, we read the document, increment the value by one, and write the document back using the Index API. But this approach requires two separate calls to ES and an entire document transfer (and back) resulting in slower performance and higher load on the cluster.
But we can do this in one API call and minimal data transfer by leveraging Painless, a scripting language for ES that can be used for aggregations, complex conditionals, and data transformations.
Painless provides us a way to write a “script” and pass it through the Update API making the entire operation efficient (for us). A partial update on the “total_views” attribute would look something like this
ctx._source.total_views += 1So, almost always, leverage the Painless scripting language of ES to make your application logic simpler. More importantly, reducing that one extra read API call and two document transfers, between ES to the client and back. ES clusters are notorious for long GC pauses and high JVM memory pressure, try to keep it to a minimum.
Although partial updates using Painless make it easy for us (the consumers of ES) to update the document, this does not mean it is efficient. Under the hood, it is still expensive (which I will discuss in the next post).
ps: admissions for my Oct cohort are open, if you are looking for a no-fluff and highly practical system design course, checkout
arpitbhayani.me/course