Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53183

[6.6.5 build 10106] - Secondary Index rollback to zero after KV node auto failover

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Steps to reproduce:

      1. Create a 6 node cluster 3KV-2index-1query.
      2. Create about 10 buckets with 2 indexes on each bucket.
      3. Load the docs until all of the buckets have data greater than memory.
      4. Kill memcached (either by OOM kill or by kill -9) and check that 1 KV node auto fails over.
      5. Do a delta recovery and rebalance. 

      Doesn't always rollback to zero. I've attached the logs for the time it did. Excerpts from the logs 

       

      [root@localhost ~]# cat /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep rollbackAllToZero
      2022-07-29T03:02:50.535-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test10 

      cbcollect 

      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.15.zip
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.17.zip
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.16.zip
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.19.zip
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.22.zip
      s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.121.215.zip 

      cc Ritam Sharma Balakumaran Gopal Dave Rigby 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.2.0-5000 contains indexing commit cafb1db with commit message:
            MB-53183 Do not check snap alignment for INIT_STREAM

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5000 contains indexing commit cafb1db with commit message: MB-53183 Do not check snap alignment for INIT_STREAM

            Build couchbase-server-7.2.0-5000 contains indexing commit ec7caf8 with commit message:
            MB-53183 Init snapAligned flag to false for timestamp

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5000 contains indexing commit ec7caf8 with commit message: MB-53183 Init snapAligned flag to false for timestamp

            Build couchbase-server-7.2.0-5000 contains indexing commit 3a4885b with commit message:
            MB-53183 Log if the disk snapshot timestamp fails snapAligned check

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5000 contains indexing commit 3a4885b with commit message: MB-53183 Log if the disk snapshot timestamp fails snapAligned check
            pavan.pb Pavan PB added a comment -

            Hi Varun Velamuri, Deepkaran Salooja this was validated on 6.6.5 build and I just checked and there's no backport commit that's been pushed for this bug fix on mad-hatter branch for 6.6.6 explicitly. Does this mean no extra validation needs to be done? Or do you think I should revalidate it on 6.6.6? Please do let me know.

            pavan.pb Pavan PB added a comment - Hi Varun Velamuri , Deepkaran Salooja this was validated on 6.6.5 build and I just checked and there's no backport commit that's been pushed for this bug fix on mad-hatter branch for 6.6.6 explicitly. Does this mean no extra validation needs to be done? Or do you think I should revalidate it on 6.6.6? Please do let me know.
            pavan.pb Pavan PB added a comment -

            There's nothing to be validated for 6.6.6 on this since no fix has gone in. It was already validated on mad-hatter (6.6.5) branch. Keeping this ticket open since 7.2 and 8.0 builds are also tied to this ticket.

            cc Chanabasappa Ghali Ashish Chaudhari Hemant Rajput

            pavan.pb Pavan PB added a comment - There's nothing to be validated for 6.6.6 on this since no fix has gone in. It was already validated on mad-hatter (6.6.5) branch. Keeping this ticket open since 7.2 and 8.0 builds are also tied to this ticket. cc Chanabasappa Ghali Ashish Chaudhari Hemant Rajput

            People

              pavan.pb Pavan PB
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty