Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53250

[Backport MB-53183 to 7.0.5] Secondary Index rollback to zero after KV node auto failover

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Steps to reproduce:

      1. Create a 6 node cluster 3KV-2index-1query.
      2. Create about 10 buckets with 2 indexes on each bucket.
      3. Load the docs until all of the buckets have data greater than memory.
      4. Kill memcached (either by OOM kill or by kill -9) and check that 1 KV node auto fails over.
      5. Do a delta recovery and rebalance. 

      Doesn't always rollback to zero. I've attached the logs for the time it did. Excerpts from the logs 

       

      [root@localhost ~]# cat /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep rollbackAllToZero
      		 
      2022-07-29T03:02:50.535-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test10

      cbcollect 

      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.15.zip
      		 
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.17.zip
      		 
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.16.zip
      		 
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.19.zip
      		 
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.22.zip
      		 
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.121.215.zip

      cc Ritam Sharma Balakumaran Gopal Dave Rigby 

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-53183 [6.6.5 build 10106] - Secondary Index rollback to zero after KV node auto failover

          • Major - Major loss of function.
          • Resolved
          For Gerrit Dashboard: MB-53250
          # Subject Branch Project Status CR V
          181851,4 MB-53250 Init snapAligned flag to false for timestamp cheshire-cat indexing Status: MERGED +2 +1
          181852,6 MB-53250 Do not check snap alignment for INIT_STREAM cheshire-cat indexing Status: MERGED +2 +1
          181995,1 MB-53250 Fix build failure cheshire-cat indexing Status: MERGED +2 +1

          Activity

            People

              hemant.rajput Hemant Rajput
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty