Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52589

[BP of MB-52588]: [System Test] Rebalance exited due to 1200 second timeout

    XMLWordPrintable

Details

    Description

      QE TEST

      -test tests/eventing/neo/test_eventing_rebalance.yml -scope tests/eventing/neo/scope_eventing_rebalance.yml

      Day - 1
      Cycle - 1
      Scale - 3
      This run involves changes made to the component system test to incorporate multi collection feature.

      TEST STEP
      Rebalance in pair of kv nodes and single eventing node.

      [2022-06-16T07:17:01-07:00, sequoiatools/couchbase-cli:7.1:d0846b] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.17 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2022-06-16T07:17:19-07:00, sequoiatools/couchbase-cli:7.1:1e92c2] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.18 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2022-06-16T07:17:38-07:00, sequoiatools/couchbase-cli:7.1:d8bf59] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.23 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      		 
      [2022-06-16T07:17:49-07:00, sequoiatools/couchbase-cli:7.1:10a87b] rebalance -c 172.23.104.16:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs 10a87b
      		 
      docker start 10a87b
      		 
       
      		 
      *Unable to display progress bar on this os
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.

      REBALANCE FAILURE

      2022-06-16T08:08:09.837-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.104.16) - Rebalance exited with reason {service_rebalance_failed,eventing,
      		 
                                    {worker_died,
      		 
                                     {'EXIT',<0.24675.105>,
      		 
                                      {rebalance_failed,
      		 
                                       {service_error,
      		 
                                        <<"eventing rebalance hasn't made progress for past 1200 secs">>}}}}}.
      		 
      Rebalance Operation Id = 43b8ed3aac955249318847e9ec4f92c8

      Vb shuffle did not complete for 2 out of the 3 eventing nodes involved.

      2022-06-16T08:07:39.700-07:00 [Info] util::GetProgress endpointURL: http://172.23.104.21:8096/getRebalanceProgress VbsRemainingToShuffle: 12 VbsOwnedPerPlan: 456
      		 
      2022-06-16T08:07:39.732-07:00 [Info] util::GetProgress endpointURL: http://172.23.104.23:8096/getRebalanceProgress VbsRemainingToShuffle: 0 VbsOwnedPerPlan: 0
      		 
      2022-06-16T08:07:39.769-07:00 [Info] util::GetProgress endpointURL: http://172.23.96.31:8096/getRebalanceProgress VbsRemainingToShuffle: 13 VbsOwnedPerPlan: 341
      		 
      2022-06-16T08:07:39.771-07:00 [Info] util::GetProgress endpointURL: http://127.0.0.1:8096/getAggRebalanceProgress VbsRemainingToShuffle: 25 VbsOwnedPerPlan: 797

      Attachments

        Issue Links

          Clones

          Bug - A problem which impairs or prevents the functions of the product. MB-52588 [System Test] Rebalance exited due to 1200 second timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sujay.gad Sujay Gad
              ankit.prabhu Ankit Prabhu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty