Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60056

Rebalance exited with reason {service_rebalance_failed,index, {agent_died,<36871.17091.403>,noconnection}}.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 7.6.0
    • 7.6.0
    • ns_server
    • 7.6.0-1899

    Description

      Step To Recreate:

      1. Create a 4 node cluster( 2 KV((172.23.110.64, 172.23.110.66) and 2 index+n1ql (172.23.110.65, 172.23.110.67) nodes
      2. Create a bucket with storage backend == magma (replicas=1)
      3. Load 100000000 docs(doc size =1024 bytes)
      4. Load travel sample and gamesim-sample
      5. Flush travel sample docs
      6. Update all the items
      7. Create 100000000 new docs(doc szie == 1024 bytes)
      8. Update all the newly create docs
      9. Start new data load(create:delete:upsert:read)
      10. While data loading is going on Rebalance in a KV node(new node coming in=='172.23.110.127')
      11. While data loading is going on Rebalance out a KV node(node going out == 172.23.110.127)
      12. While data loading is going on Rebalance in a indexing+n1ql node(172.23.110.27) (Faster rebalance was not enabled)
      13. Rebalance failed with below error

      Rebalance Error:

      Rebalance exited with reason {service_rebalance_failed,index,
      {agent_died,<36871.17091.403>,noconnection}}.
      Rebalance Operation Id = 4e15338ea6c3e36970b46d060157ecb3
      

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=1,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=22,assert_crashes_on_load=True,num_collections=51,maxttl=10,num_indexes=10,pc=10,indexer_mem_quota=0,index_nodes=2,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey'
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty