Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60665

[Upgrade] : CBAS rebalance out hung in a mixed mode cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • None
    • 7.6.0
    • analytics
    • Operating System : Debian GNU/Linux 10 (buster)
      Initial Version : Couchbase Enterprise Edition build 7.1.0-2556
      Upgrade Version : Couchbase Enterprise Edition build 7.6.0-2095

    Description

      Steps to reproduce

      1. Created a 5 node cluster with Couchbase Enterprise Edition build 7.1.0-2556
        1. 172.23.123.17 - cbas
        2. 172.23.122.227 - cbas
        3. 172.23.123.15 - cbas
        4. 172.23.123.1 - kv, index, n1ql
        5. 172.23.123.14 - kv, index, n1ql
      2. Created a couchstore bucket named "bucket-1" and loaded 10000 documents onto it
      3. Created a few dataverses, links, datasets, synonyms, index and secondary indexes on cbas
      4. Rebalanced out node 172.23.122.227 and upgraded to Couchbase Enterprise Edition build 7.6.0-2095 and added it back
      5. Rebalanced out node 172.23.123.1 and upgraded to Couchbase Enterprise Edition build 7.6.0-2095 and added it back
      6. In this mixed mode rebalanced out node 172.23.123.15  

      Rebalance has hung for 30+ minutes

      Observing a weird per node progress in /pools/default/tasks

      "perNode": {"ns_1@172.23.123.1": {"progress": 66.66666666666666},"ns_1@172.23.123.14": {"progress": 66.66666666666666},"ns_1@172.23.123.15": {"progress": 2.544999999999963e-11},"ns_1@172.23.122.227": {"progress": 2.544999999999963e-11},"ns_1@172.23.123.17": {"progress": 2.544999999999963e-11}},

       

      TAF Script to reproduce

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-analytics-vset00-00-analytics_upgrade_from_7.1.0_with_collections/testexec.23795.ini -p GROUP=7_1_0,kv_quota_percent=70,bucket_storage=couchstore,key=test_collections,get-cbcollect-info=True,upgrade_version=7.6.0-2095,aws_access_key=xxxxxx,aws_secret_key=xxxxxx,sirius_url=http://172.23.120.103:4000 -t upgrade.cbas_upgrade.UpgradeTests.test_upgrade,upgrade_chain=7.1.0,upgrade_type=online_incremental,update_nodes=kv;cbas,nodes_init=5,services_init=kv:index:n1ql-kv:index:n1ql-cbas-cbas-cbas,pre_update_no_of_dv=2,pre_update_ds_per_dv=4,pre_update_no_of_synonym=5,pre_update_no_of_index=3,replica_num=3,override_spec_params=num_buckets;num_scopes;num_collections;replicas;num_items,num_items=10000,num_buckets=3,num_scopes=5,num_collections=5,no_of_dv=10,ds_per_dv=3,no_of_synonym=10,no_of_index=5,GROUP=7_1_0'

      Job name : debian-analytics-analytics_upgrade_from_7.1.0_with_collections

      Job ref : https://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-2095/jenkins_logs/test_suite_executor-TAF/309106/

       

      Attachments

        Issue Links

          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. MB-57870 [System Test] HYR0105: Cannot drop in-use index error seen in analytics logs

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty