Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52589

[BP of MB-52588]: [System Test] Rebalance exited due to 1200 second timeout

    XMLWordPrintable

Details

    Description

      QE TEST

      -test tests/eventing/neo/test_eventing_rebalance.yml -scope tests/eventing/neo/scope_eventing_rebalance.yml
      

      Day - 1
      Cycle - 1
      Scale - 3
      This run involves changes made to the component system test to incorporate multi collection feature.

      TEST STEP
      Rebalance in pair of kv nodes and single eventing node.

      [2022-06-16T07:17:01-07:00, sequoiatools/couchbase-cli:7.1:d0846b] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.17 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2022-06-16T07:17:19-07:00, sequoiatools/couchbase-cli:7.1:1e92c2] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.18 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2022-06-16T07:17:38-07:00, sequoiatools/couchbase-cli:7.1:d8bf59] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.23 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing
      [2022-06-16T07:17:49-07:00, sequoiatools/couchbase-cli:7.1:10a87b] rebalance -c 172.23.104.16:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
       
      docker logs 10a87b
      docker start 10a87b
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      REBALANCE FAILURE

      2022-06-16T08:08:09.837-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.104.16) - Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.24675.105>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 1200 secs">>}}}}}.
      Rebalance Operation Id = 43b8ed3aac955249318847e9ec4f92c8
      

      Vb shuffle did not complete for 2 out of the 3 eventing nodes involved.

      2022-06-16T08:07:39.700-07:00 [Info] util::GetProgress endpointURL: http://172.23.104.21:8096/getRebalanceProgress VbsRemainingToShuffle: 12 VbsOwnedPerPlan: 456
      2022-06-16T08:07:39.732-07:00 [Info] util::GetProgress endpointURL: http://172.23.104.23:8096/getRebalanceProgress VbsRemainingToShuffle: 0 VbsOwnedPerPlan: 0
      2022-06-16T08:07:39.769-07:00 [Info] util::GetProgress endpointURL: http://172.23.96.31:8096/getRebalanceProgress VbsRemainingToShuffle: 13 VbsOwnedPerPlan: 341
      2022-06-16T08:07:39.771-07:00 [Info] util::GetProgress endpointURL: http://127.0.0.1:8096/getAggRebalanceProgress VbsRemainingToShuffle: 25 VbsOwnedPerPlan: 797
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.2-3321 contains eventing commit d114e65 with commit message:
            MB-52589: Add manifest id while recreating checkpoint blob

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.2-3321 contains eventing commit d114e65 with commit message: MB-52589 : Add manifest id while recreating checkpoint blob
            sujay.gad Sujay Gad added a comment -

            Issue not observed with component system test run on 7.1.2-3321.

            sujay.gad Sujay Gad added a comment - Issue not observed with component system test run on 7.1.2-3321.

            Build couchbase-server-7.2.0-5000 contains eventing commit d114e65 with commit message:
            MB-52589: Add manifest id while recreating checkpoint blob

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5000 contains eventing commit d114e65 with commit message: MB-52589 : Add manifest id while recreating checkpoint blob

            People

              sujay.gad Sujay Gad
              ankit.prabhu Ankit Prabhu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty