Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49877

[Magma 10TB, 1%]: memcached cannot accept new connections due to deadlock in Magma::GetStats

    XMLWordPrintable

Details

    Description

      1. Step 1: Create a 4 node cluster
      2. Step 2: Create required buckets and collections.
      3. Step 3: Create 625000 items sequentially
      4. Step 4: Update 625000 RandonKey keys to create 50 percent fragmentation
      5. Step 5: Create 625000 items sequentially
      6. Step 6: Update 625000 RandonKey keys to create 50 percent fragmentation
      7. Step 7: Rebalance in with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      8. Step 8: Crash Magma/memc with Loading of docs
      9. Step 9: Rebalance Out with Loading of docs. Abort and resume rebl at 20%. Rebalance failed on resumption.

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.8491.142>,
      {{{bad_return_value,{error,timeout}},
      {gen_server,call,
      ['ns_memcached-GleamBookUsers0',
      {delete_vbucket,66},
      360000]}},
      {gen_server,call,
      [{'janitor_agent-GleamBookUsers0',
      'ns_1@172.23.110.65'},
      {if_rebalance,<0.8373.142>,
      {wait_dcp_data_move,
      ['ns_1@172.23.110.64',
      'ns_1@172.23.110.66'],
      406}},
      infinity]}}}}}.
      Rebalance Operation Id = ab0fe02c379c617b89c2ca96499125e2
      

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'
      

      Attachments

        1. gdb_session_useful_stats_dump.txt
          23 kB
          Rohan Suri
        2. lsmtreestats_page1.png
          2.14 MB
          Rohan Suri
        3. lsmtreestats_page2.png
          2.29 MB
          Rohan Suri
        4. magma_dump_kvstore245.txt
          26 kB
          Rohan Suri
        5. MB-49877_gdb_session.txt
          61 kB
          Dave Rigby
        6. MB-49877_gdb_thread_apply_all_bt_full.txt
          5.18 MB
          Dave Rigby
        7. Rebalance Failed during vB movement.png
          202 kB
          Ritesh Agarwal
        8. TestLogs.txt
          5.22 MB
          Ritesh Agarwal
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty