How do updates work under the hood in Elasticsearch? ⚡
Elasticsearch uses Lucene to power the full-text search and lookups, and because Lucene stores data in immutable segments, ES cannot do in-place updates which makes the updates expensive.
Whenever an update reaches the engine, it has to reindex the entire document, regardless of which API is used (full or partial updates). Upon receiving the request, new Lucene segments are created and older versions of the affected documents (if any) are soft deleted. An update operation is thus an insert with some soft deletes.
Soft deletion means that the data would be lying around bloating the storage and memory of the index. Merge and compaction would spike up the garbage collection operations, putting a toll on the memory, CPU, and cluster performance.
So, if you an update heavy ES use case, do tune your cluster properly; some parameters to consider are refresh interval, shard allocation, and gc parameters like max memory, pool size, and algorithm. One thing that worked for me was to rotate ES infrastructure periodically.
⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
youtube.com/c/ArpitBhayani