Ever wonder how replication happens between Master and Replica? How changes on Master propagates to Replica?
This is a short summary of how it happens, the detailed essay is in the comments.
Any write operation happening on the Master is logged in the Replication log file as an event. The format in which these events are logged in the Log file is called Replication Format. The two Replication formats that are widely used across distributed data stores are Statement-based and Row-based formats.
✨ Statement-based Format The Master records the operation as an event in its log, and when the Replica reads this log, it executes the same operation on its copy of data. This way, the operation on the Master is executed on the Replica, which keeps it in sync with the Master.
So
UPDATE tasks SET is_done = true WHERE user_id = 53;
is logged as
UPDATE tasks SET is_done = true WHERE user_id = 53;
👉 Advantages:
- Smaller log files
- Log files can be used to audit the database
👉 Disadvantages
- Non-deterministic operations like RAND(), UUID(), will yield different values on Master and Replica
- The Replica lag depends on the load and concurrent queries executing during replication.
✨ Row-based Format In Row-based format, the Master logs the updates on the individual data item instead of the operation. So the entry made in the Log file would indicate how the data has changed on the Master. Hence, when the Replica reads this log, it updates its copy of the data by applying the changes on its data items. This way, the operation on the Master happens on the Replica, and the Replica remains in sync with the Master.
So,
UPDATE tasks SET is_done = true WHERE user_id = 53;
is logged as
tasks:121 is_done=true tasks:142 is_done=true tasks:643 is_done=true tasks:713 is_done=true tasks:862 is_done=true
👉 Advantages
- changes can be safely and predictably applied on the Replica
- locks are fewer and shorter
👉 Disadvantages
- If an operation affects 5000 rows, the Master would create 5000 entries in the log file
- longer lock taken during logging affects the throughput
To date, I have written ~60 articles on Distributed Systems, System Design, Advanced Algorithms, and Python Internals. Right now I am running a series on Distributed Systems and System Design, which is shared through my newsletter.
1800+ people have subscribed to my newsletter. Join them at arpitbhayani.me/newsletter.
If you found this helpful, do LIKE, SHARE, and SPREAD THE WORD ✨
If you want to master System Design and learn it the right way, you can definitely enroll in my upcoming cohort starting September 4th.
We will together build some of the most amazing systems and dissect them to understand the intricate details. You can find the week-by-week curriculum and topics, benefits, testimonials, and other details 👉 https://lnkd.in/dtBk7eE.