Details
Description
The janitor exists to correct the state of the world and it runs periodically to check that the state is consistent with its understanding of what the state should be.
When the janitor detects that there's a difference between what is and what should be it does the following:
- Removes undesired replications and adds desired replications
- Adjusts the state of vbuckets to be correct.
In a failed rebalance, it can be the case that a vbucket state needs to be changed from DEAD -> ACTIVE. Because janitor makes changes in the order stated, new replications from the about-to-be-activated vbucket are created while it is still officially dead. This works right now but there's a companion issue filed against KV engine which would prevent this from happening (MB-17230). It seems that it would be safer and more correct to change janitor into a 3 step process:
- Remove undesired replications
- Adjust vbucket state
- Add desired replications