Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60854

[Upgrade] : Analytics rebalance hangs post upgrade to 7.6

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.6.0
    • 7.6.0
    • analytics
    • Operating System : Debian
      Initial Version : Couchbase Enterprise Edition build 7.1.1-3175
      Upgrade Version : Couchbase Enterprise Edition build 7.6.0-2149

    Description

      Steps to reproduce

      Steps to reproduce

      1. Created a cluster on Couchbase Enterprise Edition build 7.1.1-3175 with the following setup 
        1. 172.23.121.136 - cbas
        2. 172.23.121.174 - index, kv, n1ql
        3. 172.23.121.194 - index, kv, n1ql
        4. 172.23.121.135 - cbas 
        5. 172.23.121.198 - cbas 
      2. Created a bucket called "bucket-0" 
      3. Loaded 10000 items onto it
      4. Created dataverses, links, datasets, synonyms, indexes 
      5. Upgraded the whole cluster to 7.6.0-2149 by failing over a node and then upgrade and add back
      6. Started a rebalance post upgrade - Rebalance hangs

       

      Response of /pools/default/tasks

       

      [{"statusId":"2d35895d90e955e571aaa80b3fe952f9","type":"rebalance","subtype":"rebalance","recommendedRefreshPeriod":0.25,"status":"running","progress":26.66666666670339,"perNode":{"ns_1@172.23.121.194":{"progress":66.66666666666666},"ns_1@172.23.121.174":{"progress":66.66666666666666},"ns_1@172.23.121.135":{"progress":6.119999999999877e-11},"ns_1@172.23.121.136":{"progress":6.119999999999877e-11},"ns_1@172.23.121.198":{"progress":6.119999999999877e-11}},"detailedProgress":{"bucket":"bucket-2","bucketNumber":1,"bucketsCount":1,"perNode":{"ns_1@172.23.121.198":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.194":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.174":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.136":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}},"ns_1@172.23.121.135":{"ingoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0},"outgoing":{"docsTotal":0,"docsTransferred":0,"activeVBucketsLeft":0,"replicaVBucketsLeft":0}}}},"stageInfo":{"analytics":{"totalProgress":6.119999999999875e-11,"perNodeProgress":{"ns_1@172.23.121.135":6.119999999999876e-13,"ns_1@172.23.121.136":6.119999999999876e-13,"ns_1@172.23.121.198":6.119999999999876e-13},"startTime":"2024-02-19T03:27:36.191-08:00","completedTime":false,"timeTaken":6130726},"index":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:30.139-08:00","completedTime":"2024-02-19T03:27:36.191-08:00","timeTaken":6052},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:29.420-08:00","completedTime":"2024-02-19T03:27:30.133-08:00","timeTaken":713},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.121.194":1,"ns_1@172.23.121.174":1},"startTime":"2024-02-19T03:27:30.133-08:00","completedTime":"2024-02-19T03:27:30.139-08:00","timeTaken":6}},"rebalanceId":"3d6830874371afe45a43f9a5afaeea70","nodesInfo":{"active_nodes":["ns_1@172.23.121.136","ns_1@172.23.121.135","ns_1@172.23.121.198","ns_1@172.23.121.174","ns_1@172.23.121.194"],"keep_nodes":["ns_1@172.23.121.136","ns_1@172.23.121.135","ns_1@172.23.121.198","ns_1@172.23.121.174","ns_1@172.23.121.194"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.121.136"}]

      The progress for nodes ns_1@172.23.121.135, ns_1@172.23.121.136 and ns_1@172.23.121.198 is weirdly stuck at 6.119999999999877e-11

       

       

      "ns_1@172.23.121.135": {"progress": 6.119999999999877e-11},"ns_1@172.23.121.136": {"progress": 6.119999999999877e-11},"ns_1@172.23.121.198": {"progress": 6.119999999999877e-11}

       

      Marking this is a regression since this was not seen in runs for RC4 - 7.6.0-2119

       

       

      TAF Script to reproduce

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-analytics-vset00-00-analytics_upgrade_with_failover_from_7.1.1_with_collections/testexec.24941.ini -p GROUP=7_1_1;failover_upgrade,kv_quota_percent=70,bucket_storage=couchstore,key=test_collections,get-cbcollect-info=True,upgrade_version=7.6.0-2151,aws_access_key=xxxxx,aws_secret_key=xxxxx,sirius_url=http://172.23.120.103:4000 -t upgrade.cbas_upgrade.UpgradeTests.test_upgrade_with_failover,upgrade_chain=7.1.1,upgrade_type=failover_delta_recovery,update_nodes=kv;cbas,nodes_init=5,services_init=kv:index:n1ql-kv:index:n1ql-cbas-cbas-cbas,pre_update_no_of_dv=2,pre_update_ds_per_dv=4,pre_update_no_of_synonym=5,pre_update_no_of_index=3,replica_num=3,override_spec_params=num_buckets;num_scopes;num_collections;replicas;num_items,num_items=10000,num_buckets=3,num_scopes=5,num_collections=5,no_of_dv=10,ds_per_dv=3,no_of_synonym=10,no_of_index=5,GROUP=7_1_1;failover_upgrade,cbas_cc_node_upgrade_sequence=first'

      Job name : debian-analytics_analytics_upgrade_with_failover_from_7.1.1_with_collections

      Job ref : http://qa.sc.couchbase.com/job/test_suite_executor-TAF/313214/console

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-60840 Intermittent failure during authentication handshake failed initial rebalance

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty