Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40251

[Magma] - Error occured during memtable flush (IOError

    XMLWordPrintable

Details

    Description

      Script to Repro

      ./testrunner -i /tmp/win10-bucket-ops.ini rerun=False,crash_warning=True,quota_percent=95 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,quota_percent=80,GROUP=rebalance_with_collection_crud

      Basically we add 2 nodes and remove 1 node and do a rebalance. It fails with the crash with following bt. Interestingly even though bt has full of DurabilityMonitor, this test was run with non durable writes.

      bt full
      bt_full.txt
       
      On a side note, also saw a few other interesting messages like the ones shown below.

      [root@s71920 logs]# grep 'segment delete failed error:IOError' *
      		 
      babysitter.log:memcached<0.115.0>: 2020-07-01T09:48:16.323573-07:00 CRITICAL WAL::Trim 18:6070272 segment delete failed error:IOError: Error (2) removing segment /opt/couchbase/var/lib/couchbase/data/bucket1/magma.3/wal/wal.17 - No such file or directory
      		 
      memcached.log.000269.txt:2020-07-01T09:48:14.375924-07:00 CRITICAL WAL::Trim 18:6676480 segment delete failed error:IOError: Error (2) removing segment /opt/couchbase/var/lib/couchbase/data/bucket1/magma.0/wal/wal.17 - No such file or directory
      		 
      memcached.log.000269.txt:2020-07-01T09:48:16.323573-07:00 CRITICAL WAL::Trim 18:6070272 segment delete failed error:IOError: Error (2) removing segment /opt/couchbase/var/lib/couchbase/data/bucket1/magma.3/wal/wal.17 - No such file or directory
      		 
      grep: rebalance: Is a directory
      		 
      [root@s71920 logs]#

      cbcollect_info attached.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-40179 Failed to write tree state, Error occured during memtable flush. Files/vBuckets delete are not is sync with data flush during rebalance.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-40257 [Collections] - memcached crashes in DurabilityMonitor::SyncWrite::getBySeqno

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          For Gerrit Dashboard: MB-40251
          # Subject Branch Project Status CR V
          131920,5 MB-40251 kvstore: Make Destroy API to block wait until data is removed master magma Status: MERGED +2 +1
          131949,2 Revert "MB-40251 kvstore: Make Destroy API to block wait until data is removed" master magma Status: MERGED +2 +1
          132040,3 MB-40251 kvstore: Block wait for data removal in DeleteKVStore master magma Status: MERGED +2 +1

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty