Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45468

[Windows] - Rebalance fails with "Unable to read index layout from cluster 127.0.0.1:8091. err = Invalid couchbase-server version"

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • No

    Description

      Scripts to Repro

      ./testrunner -i /tmp/win10-bucket-ops-11111222223334444.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_after_ops,nodes_out=1,replicas=1,items=10000,GROUP=OUT;P0    incremental_rebalance_out_with_ops,replicas=2,items=0,standard_buckets=2,sasl_buckets=2,standard_bucket_priority=low:high,sasl_bucket_priority=low:high,value_size=1024,GROUP=OUT;P0
      

      ./testrunner -i /tmp/win10-bucket-ops-11111222223334444.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True -t failover.AutoFailoverTests.AutoFailoverTests.test_autofailover,timeout=5,num_node_failures=1,failover_orchestrator=True,failover_action=stop_server,nodes_init=3
      

      Saw this failure on the above 2 tests on window runs.

      We are trying to create a cluster when rebalance fails as shown below.

      [2021-04-02 01:30:10,497] - [basetestcase:2388] INFO - **** add 'admin' role to 'cbadminbucket' user ****
      [2021-04-02 01:30:10,519] - [basetestcase:332] INFO - done initializing cluster
      [2021-04-02 01:30:11,334] - [task:775] INFO - adding node 172.23.136.109:8091 to cluster
      [2021-04-02 01:30:11,334] - [rest_client:1487] INFO - adding remote node @172.23.136.109:8091 to this cluster @172.23.136.103:8091
      [2021-04-02 01:30:21,353] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds 
      [2021-04-02 01:30:21,354] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
      [2021-04-02 01:30:36,582] - [task:775] INFO - adding node 172.23.136.111:8091 to cluster
      [2021-04-02 01:30:36,582] - [rest_client:1487] INFO - adding remote node @172.23.136.111:8091 to this cluster @172.23.136.103:8091
      [2021-04-02 01:30:46,601] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds 
      [2021-04-02 01:30:46,602] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
      [2021-04-02 01:31:01,480] - [task:775] INFO - adding node 172.23.136.112:8091 to cluster
      [2021-04-02 01:31:01,480] - [rest_client:1487] INFO - adding remote node @172.23.136.112:8091 to this cluster @172.23.136.103:8091
      [2021-04-02 01:31:11,500] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds 
      [2021-04-02 01:31:11,500] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
      [2021-04-02 01:31:26,303] - [task:775] INFO - adding node 172.23.136.115:8091 to cluster
      [2021-04-02 01:31:26,303] - [rest_client:1487] INFO - adding remote node @172.23.136.115:8091 to this cluster @172.23.136.103:8091
      [2021-04-02 01:31:36,323] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds 
      [2021-04-02 01:31:36,323] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
      [2021-04-02 01:31:56,092] - [task:775] INFO - adding node 172.23.136.114:8091 to cluster
      [2021-04-02 01:31:56,093] - [rest_client:1487] INFO - adding remote node @172.23.136.114:8091 to this cluster @172.23.136.103:8091
      [2021-04-02 01:32:06,116] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds 
      [2021-04-02 01:32:06,117] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
      [2021-04-02 01:32:24,630] - [rest_client:1714] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.136.103,ns_1@172.23.136.109,ns_1@172.23.136.111,ns_1@172.23.136.112,ns_1@172.23.136.114,ns_1@172.23.136.115', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'}
      [2021-04-02 01:32:24,685] - [rest_client:1719] INFO - rebalance operation started
      [2021-04-02 01:32:24,689] - [rest_client:1881] INFO - rebalance percentage : 0.00 %
      [2021-04-02 01:32:24,689] - [task:844] INFO - Rebalance - status: running, progress: 0.00%
      [2021-04-02 01:32:34,727] - [rest_client:1864] ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
      [2021-04-02 01:32:34,747] - [rest_client:3816] INFO - Latest logs from UI on 172.23.136.103:
      [2021-04-02 01:32:34,748] - [rest_client:3817] ERROR - {'node': 'ns_1@172.23.136.103', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1617352347001, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n                              {worker_died,\n                               {\'EXIT\',<0.8358.15>,\n                                {rebalance_failed,\n                                 {service_error,\n                                  <<"Unable to read index layout from cluster 127.0.0.1:8091. err = Invalid couchbase-server version">>}}}}}.\nRebalance Operation Id = e7e14713eb3a0bc171c1d5cf28236f1e', 'serverTime': '2021-04-02T01:32:27.001Z'}
      

      cbcollect_info attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              steve.watanabe Steve Watanabe
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty