ElasticSearch is a pretty popular search engine, but what happens when we issue an update to a document? It is not as straightforward as you’d think…
Elasticsearch uses Lucene to power the full-text search and lookups, and because Lucene stores data in immutable segments, ES cannot do in-place updates which makes the updates expensive.
Whenever an update reaches the engine, it has to reindex the entire document, regardless of which API is used (full or partial updates). Upon receiving the request, new Lucene segments are created, and older versions of the affected documents (if any) are soft deleted. An update operation is thus an insert with some soft deletes.
Soft deletion means that the data would be lying around bloating the storage and memory of the index. Merge and compaction would spike up the garbage collection operations, putting a toll on the memory, CPU, and cluster performance.
So, if you an update heavy ES use case, do tune your cluster properly; some parameters to consider are refresh interval, shard allocation, and gc parameters like max memory, pool size, and algorithm. One thing that worked for me was to rotate the ES infrastructure periodically.
I suffered a ton because of this while I was building search, had to dig deep to understand why ES cluster eats so much memory :)
btw, enrollments open for sys design June cohort - arpitbhayani.me/course