Elasticsearch document life cycle

An Elasticsearch cluster can have multiple nodes. Each node contains multiple shards. In general, an Elasticsearch index will span across multiple shards residing in multiple nodes. A shard is typically a Lucene index. A Lucene index is divided into multiple segments which are more like mini indices holding data structures like inverted index, stored fields and doc values.

Lets look at what happens when each of the CRUD (Create, Read, Update Delete) operations are performed on an Elasticsearch index.

When a document is CREATED in an Elasticsearch index, then the document is said to be stored in one of the segments inside a Lucene Index. Every document that is indexed is fed into a new lucene segment and then the document is available for search. Over time, these segments keep growing with the increase in the number of new documents created. So, Elasticsearch merges these segments after certain interval. This merge operation is high memory intensive and might arise performance problems.

Depending on the use case, whether Elasticsearch has to be read optimized or write optimized one has to tune this merge process to improve performance.

When a document is DELETED, there is a bitmap that is maintained in the memory where the document is marked as deleted and not actually deleted. Lucene will use this bitmap to filter out these deleted documents when there are subsequent searches happening.

During the merge process, these deleted documents are cleaned up and completely removed.

Segments in lucene are immutable and thus document update is a 2 step process. When a document is UPDATED, then there is a new document that is added with the new changes and the old document is marked as deleted. Each time a document is updated, version of the document is incremented and stored as field in document source.

Typically, a READ/SEARCH request can be forwarded to any of the nodes in a cluster. The node that receives the request becomes coordinating node. The coordinating node will be responsible to forward the request to multiple other nodes in the cluster. Before search can actually be…

Related posts:

I was trained to be apathetic towards politics because of its notoriety for corruption and the fear that I could lose my life as a vocal supporter of an unpopular party. Be that as it may, I have…

Which is not a constructive conversation. So my question: is there a way to show what is objectively bad about CSS and maybe try to fix it? Stress in this question on objectivity, so it should not be…

You have to be ready for these questions so you can give answers that present you in a positive light and allow you to sidestep those little traps. Here are four common — but tricky — interview…