Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46627

[Windows]: Eventing rebalance got stuck for more that 2.5 hours at ~79%

    XMLWordPrintable

Details

    Description

      Build: 7.0.0-5238

      Scenario:

      Rebalancing out Eventing node from the cluster with multiple services enabled.

      (Operation Id = b8f8038679f6dd16dc26c2e7eb755ba3)

      +----------------+-------------+-----------------------+---------------+--------------+
      		 
      | Nodes          | Services    | Version               | CPU           | Status       |
      		 
      +----------------+-------------+-----------------------+---------------+--------------+
      		 
      | 172.23.107.142 | eventing    | 7.0.0-5238-enterprise | 15.3928202393 | Cluster node |
      		 
      | 172.23.106.116 | backup      | 7.0.0-5238-enterprise | 6.95321744638 | Cluster node |
      		 
      | 172.23.107.127 | cbas        | 7.0.0-5238-enterprise | 2.52666666667 | Cluster node |
      		 
      | 172.23.107.129 | kv          | 7.0.0-5238-enterprise | 39.6083333333 | Cluster node |
      		 
      | 172.23.107.126 | cbas        | 7.0.0-5238-enterprise | 7.94460276986 | Cluster node |
      		 
      | 172.23.104.247 | kv          | 7.0.0-5238-enterprise | 47.6291271521 | Cluster node |
      		 
      | 172.23.105.137 | kv          | 7.0.0-5238-enterprise | 49.554159236  | Cluster node |
      		 
      | 172.23.105.1   | index, n1ql | 7.0.0-5238-enterprise | 17.5264594289 | Cluster node |
      		 
      | 172.23.105.183 | eventing    | 7.0.0-5238-enterprise | 42.136226522  | --- OUT ---> |
      		 
      | 172.23.107.131 | index, n1ql | 7.0.0-5238-enterprise | 8.54319094682 | Cluster node |
      		 
      +----------------+-------------+-----------------------+---------------+--------------+

      Observation:

      Eventing rebalance stuck around 79% and not proceeding further for 2.5 hrs.

      Also seeing failures and timeouts in the deployed Eventing function "a3_users_search"

      Note: Possible regression due to MB-46543

      Attachments

        1. Eventing_reb_hung.png
          Eventing_reb_hung.png
          443 kB
        2. eventing-consumer.exe
          1.99 MB
        3. good_handler.txt
          29 kB
        4. libcouchbase.dll
          863 kB
        5. libcouchbase.pdb
          7.33 MB
        6. stuck_handler.txt
          35 kB

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-54699 BP 6.6.6 - Repair LCB handles on AUTH error

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-54699 BP 6.6.6 - Repair LCB handles on AUTH error

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-40061 libcouchbase instances are not kept awake during idle periods

          • Major - Major loss of function.
          • Open

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty