Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46324

[130Node]: KV rebalance failed with wait_for_memcached_failed when 9 KV nodes going out and 10 coming

    XMLWordPrintable

Details

    Description

      1. Create a 80 node cluster - 40-KV, 35-2i, 5-N1QL
      2. Create required buckets and collections.
      3. Create 500000000 items sequentially
      4. Rebalance IN 9 KV nodes with Loading of docs
      5. Rebalance OUT 9 KV nodes with Loading of docs
      6. Rebalance SWAP 9 KV nodes with Loading of docs
      7. Rebalance IN 10 KV nodes and OUT 9 KV nodes with Loading of docs
      8. Rebalance OUT 10 KV nodes and IN 9 KV nodes with Loading of docs
      9. Repeat the above steps and Rebalance failed in Step 7.

      Rebalance exited with reason {pre_rebalance_janitor_run_failed,
      "GleamBookUsers0",
      {error,wait_for_memcached_failed,
      ['ns_1@172.23.123.6']}}.
      Rebalance Operation Id = bc1f7233ca9cf49be4d727d631c4f481
      

      QE Test

      git fetch "http://review.couchbase.org/TAF" refs/changes/34/151634/69 && git checkout FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=40,graceful=True,skip_cleanup=True,num_items=500000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,eviction_policy=fullEviction,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=100,maxttl=1000,num_indexes=50,pc=1,sdk_client_pool=True,max_clients=10,index_nodes=35,query_nodes=5,cbas_nodes=0,fts_nodes=0,rebl_nodes=9 -m rest'
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty