Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49877

[Magma 10TB, 1%]: memcached cannot accept new connections due to deadlock in Magma::GetStats

    XMLWordPrintable

Details

    Description

      1. Step 1: Create a 4 node cluster
      2. Step 2: Create required buckets and collections.
      3. Step 3: Create 625000 items sequentially
      4. Step 4: Update 625000 RandonKey keys to create 50 percent fragmentation
      5. Step 5: Create 625000 items sequentially
      6. Step 6: Update 625000 RandonKey keys to create 50 percent fragmentation
      7. Step 7: Rebalance in with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      8. Step 8: Crash Magma/memc with Loading of docs
      9. Step 9: Rebalance Out with Loading of docs. Abort and resume rebl at 20%. Rebalance failed on resumption.

      Rebalance exited with reason {mover_crashed,
      		 
      {unexpected_exit,
      		 
      {'EXIT',<0.8491.142>,
      		 
      {{{bad_return_value,{error,timeout}},
      		 
      {gen_server,call,
      		 
      ['ns_memcached-GleamBookUsers0',
      		 
      {delete_vbucket,66},
      		 
      360000]}},
      		 
      {gen_server,call,
      		 
      [{'janitor_agent-GleamBookUsers0',
      		 
      'ns_1@172.23.110.65'},
      		 
      {if_rebalance,<0.8373.142>,
      		 
      {wait_dcp_data_move,
      		 
      ['ns_1@172.23.110.64',
      		 
      'ns_1@172.23.110.66'],
      		 
      406}},
      		 
      infinity]}}}}}.
      		 
      Rebalance Operation Id = ab0fe02c379c617b89c2ca96499125e2

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'

      Attachments

        1. gdb_session_useful_stats_dump.txt
          23 kB
        2. lsmtreestats_page1.png
          lsmtreestats_page1.png
          2.14 MB
        3. lsmtreestats_page2.png
          lsmtreestats_page2.png
          2.29 MB
        4. magma_dump_kvstore245.txt
          26 kB
        5. MB-49877_gdb_session.txt
          61 kB
        6. MB-49877_gdb_thread_apply_all_bt_full.txt
          5.18 MB
        7. Rebalance Failed during vB movement.png
          Rebalance Failed during vB movement.png
          202 kB
        8. TestLogs.txt
          5.22 MB
        For Gerrit Dashboard: MB-49877
        # Subject Branch Project Status CR V
        166956,6 MB-49877 magma: Restore compaction state upon failure master magma Status: MERGED +2 +1
        166986,7 MB-49877 magma: Assert compactionState <= tree snapshot state master magma Status: MERGED +2 +1

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty