Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59350

KV swap-rebalance is hung forever due to DcpConsumer missing wakeup to transmit StreamRequest

    XMLWordPrintable

Details

    Description

      1. Create a 3KV, 2 GSI-N1QL node cluster.
      2. Create magma bucket and 10 collections.
      3. Create 100000000 items: CircularKey
      4. Scale OUT with Loading of docs by 1 kv node & 1 GSI node one at a time
      5. Scale OUT with Loading of docs by 1 kv node & 1 GSI node one at a time
      6. Scale IN with Loading of docs by 1 kv node & 1 GSI node one at a time
      7. Scale IN with Loading of docs by 1 kv node & 1 GSI node one at a time
      8. Scale Disk UP with Loading of docs
      9. Scale Disk DOWN with Loading of docs
      10. Scale Compute UP with Loading of docs - Vertical scaling which lead to swap rebalance of each node
      11. Scale Compute DOWN with Loading of docs - Vertical scaling which lead to swap rebalance of each node
      12. Scale Disk + Compute UP with Loading of docs - Vertical scaling which lead to swap rebalance of each node
      13. KV node swap rebalance is hung forever.

        Starting rebalance, KeepNodes = ['ns_1@svc-d-node-019.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com',
        'ns_1@svc-d-node-020.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com',
        'ns_1@svc-d-node-024.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com',
        'ns_1@svc-qi-node-022.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com',
        'ns_1@svc-qi-node-023.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com'], EjectNodes = ['ns_1@svc-d-node-021.zh5av6rszeesxl5u.sandbox.nonprod-project-avengers.com'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 6c6c808a64eabb6073e6ab6402a094af
        

      QE test

      sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/couchbase_capella_volume_2_new.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.hostedHospital.Murphy.test_rebalance,num_items=100000000,num_buckets=1,bucket_names=GleamBook,bucket_type=membase,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,maxttl=10,pc=10,gsi_nodes=2,cbas_nodes=3,fts_nodes=3,kv_nodes=3,n1ql_nodes=2,kv_disk=1500,n1ql_disk=50,gsi_disk=500,fts_disk=1500,cbas_disk=1500,kv_compute=n2-standard-16,gsi_compute=n2-standard-16,n1ql_compute=n2-standard-16,fts_compute=n2-standard-16,cbas_compute=n2-standard-16,mutation_perc=20,key_type=CircularKey,capella_run=true,services=data-index:query,rebl_services=data-index:query,max_rebl_nodes=27,provider=GCP,region=us-east1,type=PD-SSD,size=1500,ops_rate=100000,skip_teardown_cleanup=true,wait_timeout=14400,index_timeout=28800,runtype=dedicated,skip_init=false,rebl_ops_rate=10000,collections=10,expiry=true,vh_scaling=true,horizontal_scale=1,clients_per_db=10 -m rest'
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty