Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46543

Rebalance failed due to reason "service_rebalance_failed::eventing"

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • Unknown

    Description

      Build: 7.0.0-5219

      Scenario:

      • Multi-node cluster with active KV, index+n1ql, Eventing, Cbas and backup services running
      • Couchbase bucket with replica=1
      • Running continuous rebalance in/out/swap operations on different services
      • Rebalance failed during rebalancing in 2 nodes (1 KV + 1 cbas) node into the cluster with following reason,

      Rebalance exited with reason {service_rebalance_failed,eventing,
      {worker_died,
      {'EXIT',<0.6781.158>,
      {rebalance_failed,
      {service_error,
      <<"eventing rebalance hasn't made progress for past 1200 secs">>}}}}}.
      Rebalance Operation Id = 046686d0d38420f50b44cad71ec134db

      +----------------+-------------+-----------------------+---------------+--------------+
      | Nodes          | Services    | Version               | CPU           | Status       |
      +----------------+-------------+-----------------------+---------------+--------------+
      | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.5116666667 | Cluster node |
      | 172.23.106.116 | backup      | 7.0.0-5219-enterprise | 10.2583333333 | Cluster node |
      | 172.23.107.127 | eventing    | 7.0.0-5219-enterprise | 30.1311644806 | Cluster node |
      | 172.23.107.129 | kv          | 7.0.0-5219-enterprise | 22.9166666667 | Cluster node |
      | 172.23.107.126 | cbas        | 7.0.0-5219-enterprise | 45.2258538377 | Cluster node |
      | 172.23.104.247 | kv          | 7.0.0-5219-enterprise | 22.9518595179 | Cluster node |
      | 172.23.105.1   | kv          | 7.0.0-5219-enterprise | 28.6421440357 | Cluster node |
      | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.2883333333 | Cluster node |
      | 172.23.107.131 | ['cbas']    |                       |               | <--- IN ---  |
      | 172.23.105.137 | ['kv']      |                       |               | <--- IN ---  |
      +----------------+-------------+-----------------------+---------------+--------------+

      Log Snapshot: http://supportal.couchbase.com/snapshot/17697ffadd57c91f806ed0eb05fb3cb5::0

      cbcollect files:

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.104.247.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.105.1.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.105.137.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.105.183.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.106.116.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.107.126.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.107.127.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.107.129.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.107.131.zip

      https://cb-engineering.s3.amazonaws.com/eventing_reb_failure/collectinfo-2021-05-25T191017-ns_1%40172.23.107.142.zip

       

       

      Attachments

        For Gerrit Dashboard: MB-46543
        # Subject Branch Project Status CR V

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty