Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45954

Rebalance fails because of PrepareTopologyChange timeout

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      This issue is branched from MB-45869, please take logs from that ticket.

      Rebalance at 2021-04-21T04:04:51.677-07:00 fails because of PrepareTopologyChange timeout:

      2861252 [user:error,2021-04-21T08:41:40.426-07:00,ns_1@172.23.108.103:<0.23324.0>:ns_orchestrator:log_rebalance_completion:1404]Rebalance exited with reason {service_rebalance_failed,eventing,
      		 
      2861253                               {agent_died,<30247.24974.0>,
      		 
      2861254                                {linked_process_died,<30247.30085.1993>,
      		 
      2861255                                 {timeout,
      		 
      2861256                                  {gen_server,call,
      		 
      2861257                                   [<30247.25022.0>,
      		 
      2861258                                    {call,"ServiceAPI.PrepareTopologyChange",
      		 
      2861259                                     #Fun<json_rpc_connection.0.77329884>},
      		 
      2861260                                    60000]}}}}}.

      Which is a call to eventing on 172.23.123.27

      On 172.23.123.27 I see that eventing created a lot (~2000) of streaming connections against ns_server around that time, which I believe could lead to a timeout.

      Please investigate why there are so many streaming connections.

      Attachments

        1. screenshot-1.png
          119 kB
          Dave Finlay
        2. screenshot-2.png
          69 kB
          Dave Finlay
        3. Screen Shot 2021-04-26 at 5.57.36 PM.png
          344 kB
          Timofey Barmin
        4. Screen Shot 2021-04-30 at 1.03.46 PM.png
          211 kB
          Timofey Barmin
        5. Screen Shot 2021-05-21 at 10.35.51 AM.png
          330 kB
          Timofey Barmin
        6. Screen Shot 2021-05-21 at 10.37.58 AM.png
          251 kB
          Timofey Barmin
        7. Screen Shot 2021-05-21 at 10.40.46 AM.png
          313 kB
          Timofey Barmin
        8. screenshot-3.png
          39 kB
          Dave Finlay

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-45869 [System Test][Analytics] Rebalance failed with error - java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: [xxxx], state: ACTIVE)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-46782 [BP 6.6.3][System Test][Analytics] Rebalance failed with error - java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: [xxxx], state: ACTIVE)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-45954
          # Subject Branch Project Status CR V
          152530,1 MB-45954: Close gocb cluster instance on error unstable eventing Status: ABANDONED 0 0
          152544,9 MB-45954: Cache gocb cluster handles unstable eventing Status: MERGED +2 +1
          152668,5 MB-45954: Close connection when no function listening on source or metatdata bucket unstable eventing Status: MERGED +2 +1
          153182,3 MB-45954: Fix race between closing channel and reading from socket unstable eventing Status: MERGED +2 +1

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              timofey.barmin Timofey Barmin
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty