Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46627

[Windows]: Eventing rebalance got stuck for more that 2.5 hours at ~79%

    XMLWordPrintable

Details

    Description

      Build: 7.0.0-5238

      Scenario:

      Rebalancing out Eventing node from the cluster with multiple services enabled.

      (Operation Id = b8f8038679f6dd16dc26c2e7eb755ba3)

      +----------------+-------------+-----------------------+---------------+--------------+
      | Nodes          | Services    | Version               | CPU           | Status       |
      +----------------+-------------+-----------------------+---------------+--------------+
      | 172.23.107.142 | eventing    | 7.0.0-5238-enterprise | 15.3928202393 | Cluster node |
      | 172.23.106.116 | backup      | 7.0.0-5238-enterprise | 6.95321744638 | Cluster node |
      | 172.23.107.127 | cbas        | 7.0.0-5238-enterprise | 2.52666666667 | Cluster node |
      | 172.23.107.129 | kv          | 7.0.0-5238-enterprise | 39.6083333333 | Cluster node |
      | 172.23.107.126 | cbas        | 7.0.0-5238-enterprise | 7.94460276986 | Cluster node |
      | 172.23.104.247 | kv          | 7.0.0-5238-enterprise | 47.6291271521 | Cluster node |
      | 172.23.105.137 | kv          | 7.0.0-5238-enterprise | 49.554159236  | Cluster node |
      | 172.23.105.1   | index, n1ql | 7.0.0-5238-enterprise | 17.5264594289 | Cluster node |
      | 172.23.105.183 | eventing    | 7.0.0-5238-enterprise | 42.136226522  | --- OUT ---> |
      | 172.23.107.131 | index, n1ql | 7.0.0-5238-enterprise | 8.54319094682 | Cluster node |
      +----------------+-------------+-----------------------+---------------+--------------+

      Observation:

      Eventing rebalance stuck around 79% and not proceeding further for 2.5 hrs.

      Also seeing failures and timeouts in the deployed Eventing function "a3_users_search"

      Note: Possible regression due to MB-46543

      Attachments

        1. Eventing_reb_hung.png
          Eventing_reb_hung.png
          443 kB
        2. eventing-consumer.exe
          1.99 MB
        3. good_handler.txt
          29 kB
        4. libcouchbase.dll
          863 kB
        5. libcouchbase.pdb
          7.33 MB
        6. stuck_handler.txt
          35 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-5277 contains eventing commit 4b97192 with commit message:
            MB-46627:

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5277 contains eventing commit 4b97192 with commit message: MB-46627 :

            Build couchbase-server-7.0.0-5277 contains eventing commit 6a790eb with commit message:
            MB-46627 : Recreate lcb_Instance upon AUTH error during bucket op

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5277 contains eventing commit 6a790eb with commit message: MB-46627 : Recreate lcb_Instance upon AUTH error during bucket op

            Build couchbase-server-7.1.0-1008 contains eventing commit 4b97192 with commit message:
            MB-46627:

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1008 contains eventing commit 4b97192 with commit message: MB-46627 :

            Build couchbase-server-7.1.0-1008 contains eventing commit 6a790eb with commit message:
            MB-46627 : Recreate lcb_Instance upon AUTH error during bucket op

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1008 contains eventing commit 6a790eb with commit message: MB-46627 : Recreate lcb_Instance upon AUTH error during bucket op

            Ran the e2e test for more than 24hrs. without any issues.

            Validated using Enterprise Edition 7.0.0 build 5279.

            Closing this ticket.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Ran the e2e test for more than 24hrs. without any issues. Validated using Enterprise Edition 7.0.0 build 5279. Closing this ticket.

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty