Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47255

[Magma] Rebalance hung

    XMLWordPrintable

Details

    Description

      Build - 7.1.0 - 1047

      QE TEST

      eventing.eventing_rebalance.EventingRebalance.eventing.eventing_rebalance.EventingRebalance.test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations,nodes_init=6,services_init=kv-kv-eventing-eventing-eventing-index:n1ql, dataset=default, groups=simple,reset_services=True,doc-per-day=20, sock_batch_size=8, worker_count=8, cpp_worker_thread_count=8,GROUP=bucket_op_set2,get-cbcollect-info=True, bucket_storage=magma,enable_dp=True
      

      FAILURE

      2021-07-04 07:22:02 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:22:02 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:22:22 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:22:22 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:22:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:22:42 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:23:02 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:23:02 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:23:22 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:23:22 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:23:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:23:42 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:24:02 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:24:02 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:24:22 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:24:22 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:24:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:24:42 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:25:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:25:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:25:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:25:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:25:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:25:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:26:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:26:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:26:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:26:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:26:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:26:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:27:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:27:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:27:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:27:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:27:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:27:43 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:28:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:28:03 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:28:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:28:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:28:44 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:28:44 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:29:04 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:29:04 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      2021-07-04 07:29:24 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 7.00 %
      2021-07-04 07:29:24 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 7.00%
      
      

      ERROR: test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations (eventing.eventing_rebalance.EventingRebalance)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "pytests/eventing/eventing_rebalance.py", line 324, in test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations
          rebalance.result()
        File "lib/tasks/future.py", line 160, in result
          return self.__get_result()
        File "lib/tasks/future.py", line 112, in __get_result
          raise self._exception
      membase.api.exception.RebalanceFailedException: Rebalance Failed: seems like rebalance hangs. please check logs!
      

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty