Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33381

update_vbucket_map_history can lose latest maps, causing delta recovery to fail

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Currently update_vbucket_map_history , works as below

      When we get a newmap being set it checks if that map is present in the history, if so we do nothing(no reordering update the history as it was previously).

      If the map isn't present we add it to the head of the list and delete the tail if the list is above 10. 

       

      Consider a situation where we have a large number of buckets say 9, with different vbucket maps, named bucket-1 to bucket-9. The vbucket_map_history lists is full at 10 past maps. 

      On rebalance,  bucket-1 goes first it tries to set the vbucket_map_history sees that the map is already present in the history, but it is the last one on the list, we do nothing(no reordering update the history as it was previously) in this case.

      The second bucket-2 rebalance runs and we discover that it has a new map we set the map in vbucket_map_history at the expense of bucket-1's latest vbucket map. The issue here is since the vbucket_map_history is not per bucket, it is per cluster, and we don't reorder according to latest map. 

       

      So after successfully rebalance we have lost potentially bucket-1's latest vbucket map. 

       

      Assume now we failover a node and add it back for delta-recovery, we cannot recover bucket-1 because we have lost that map. 

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Abhijeeth.Nuthan Abhijeeth Nuthan created issue -
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Field Original Value New Value
            Assignee Ajit Yagaty [ ajit.yagaty ] Abhijeeth Nuthan [ abhijeeth.nuthan ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Link This issue relates to MB-33365 [ MB-33365 ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Link This issue relates to CBSE-6528 [ CBSE-6528 ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Summary update_vbucket_map_history can lose latest maps, causign delta recovery to fail update_vbucket_map_history can lose latest maps, causing delta recovery to fail
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Link This issue relates to CBSE-6528 [ CBSE-6528 ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Description Currently update_vbucket_map_history , works as below

            When we get a newmap being set it checks if that map is present in the history, if so we do nothing.

            If the map isn't present we add it to the head of the list and delete the tail if the list is above 10. 

             

            Consider a situation where we have a large number of buckets say 9, with different vbucket maps, named bucket-1 to bucket-9. The vbucket_map_history lists is full at 10 past maps. 

            On rebalance,  bucket-1 goes first it tries to set the vbucket_map_history sees that the map is already present in the history, but it is the last one on the list, we do nothing in this case.

            The second bucket-2 rebalance runs and we discover that it has a new map we set the map in vbucket_map_history at the expense of bucket-1's latest vbucket map. The issue here is since the vbucket_map_history is not per bucket, it is per cluster. 

             

            So after successfully rebalance we have lost potentially bucket-1's latest vbucket map. 

             

            Assume now we failover a node and add it back for delta-recovery, we cannot recover bucket-1 because we have lost that map. 

             
            Currently update_vbucket_map_history , works as below

            When we get a newmap being set it checks if that map is present in the history, if so we do nothing(no reordering update the history as it was previously).

            If the map isn't present we add it to the head of the list and delete the tail if the list is above 10. 

             

            Consider a situation where we have a large number of buckets say 9, with different vbucket maps, named bucket-1 to bucket-9. The vbucket_map_history lists is full at 10 past maps. 

            On rebalance,  bucket-1 goes first it tries to set the vbucket_map_history sees that the map is already present in the history, but it is the last one on the list, we do nothing(no reordering update the history as it was previously) in this case.

            The second bucket-2 rebalance runs and we discover that it has a new map we set the map in vbucket_map_history at the expense of bucket-1's latest vbucket map. The issue here is since the vbucket_map_history is not per bucket, it is per cluster. 

             

            So after successfully rebalance we have lost potentially bucket-1's latest vbucket map. 

             

            Assume now we failover a node and add it back for delta-recovery, we cannot recover bucket-1 because we have lost that map. 

             
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Description Currently update_vbucket_map_history , works as below

            When we get a newmap being set it checks if that map is present in the history, if so we do nothing(no reordering update the history as it was previously).

            If the map isn't present we add it to the head of the list and delete the tail if the list is above 10. 

             

            Consider a situation where we have a large number of buckets say 9, with different vbucket maps, named bucket-1 to bucket-9. The vbucket_map_history lists is full at 10 past maps. 

            On rebalance,  bucket-1 goes first it tries to set the vbucket_map_history sees that the map is already present in the history, but it is the last one on the list, we do nothing(no reordering update the history as it was previously) in this case.

            The second bucket-2 rebalance runs and we discover that it has a new map we set the map in vbucket_map_history at the expense of bucket-1's latest vbucket map. The issue here is since the vbucket_map_history is not per bucket, it is per cluster. 

             

            So after successfully rebalance we have lost potentially bucket-1's latest vbucket map. 

             

            Assume now we failover a node and add it back for delta-recovery, we cannot recover bucket-1 because we have lost that map. 

             
            Currently update_vbucket_map_history , works as below

            When we get a newmap being set it checks if that map is present in the history, if so we do nothing(no reordering update the history as it was previously).

            If the map isn't present we add it to the head of the list and delete the tail if the list is above 10. 

             

            Consider a situation where we have a large number of buckets say 9, with different vbucket maps, named bucket-1 to bucket-9. The vbucket_map_history lists is full at 10 past maps. 

            On rebalance,  bucket-1 goes first it tries to set the vbucket_map_history sees that the map is already present in the history, but it is the last one on the list, we do nothing(no reordering update the history as it was previously) in this case.

            The second bucket-2 rebalance runs and we discover that it has a new map we set the map in vbucket_map_history at the expense of bucket-1's latest vbucket map. The issue here is since the vbucket_map_history is not per bucket, it is per cluster, and we don't reorder according to latest map. 

             

            So after successfully rebalance we have lost potentially bucket-1's latest vbucket map. 

             

            Assume now we failover a node and add it back for delta-recovery, we cannot recover bucket-1 because we have lost that map. 

             
            raju Raju Suravarjjala made changes -
            Fix Version/s Mad-Hatter [ 15037 ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Assignee Abhijeeth Nuthan [ abhijeeth.nuthan ] Ajit Yagaty [ ajit.yagaty ]
            dfinlay Dave Finlay made changes -
            Assignee Ajit Yagaty [ ajit.yagaty ] Abhijeeth Nuthan [ abhijeeth.nuthan ]
            Abhijeeth.Nuthan Abhijeeth Nuthan made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Abhijeeth.Nuthan Abhijeeth.Nuthan made changes -
            Actual End 2019-06-25 12:39 (issue has been resolved)
            ritam.sharma Ritam Sharma made changes -
            Labels request-dev-verify
            dfinlay Dave Finlay made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              Abhijeeth.Nuthan Abhijeeth Nuthan
              Abhijeeth.Nuthan Abhijeeth Nuthan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty