Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46112

Rebalance failed with reason "agent died"

    XMLWordPrintable

Details

    Description

      Build: 7.0.0-5095

      Scenario:

      • Cluster with index,eventing,fts,backup services actively running
      • 3 FTS indexes created (2 with indexPartition=1 and 1 index with indexPartition=6)
      • Rebalance in extra FTS node into the cluster along with few other nodes with other services

      Observation:

      Fts rebalance failed with reason fts worker died,

       

      Rebalance exited with reason {service_rebalance_failed,fts,
      		 
      {worker_died,
      		 
      {'EXIT',<0.12316.37>,
      		 
      {rebalance_failed,inactivity_timeout}}}}.
      		 
      Rebalance Operation Id = b5ce1e972169e0377441008a937cb498
      		 
      ..
      		 
      ..
      		 
      Starting rebalance, KeepNodes = ['ns_1@172.23.105.175','ns_1@172.23.106.233',
      		 
      'ns_1@172.23.106.236','ns_1@172.23.106.238',
      		 
      'ns_1@172.23.106.250','ns_1@172.23.106.251',
      		 
      'ns_1@172.23.107.43','ns_1@172.23.107.44',
      		 
      'ns_1@172.23.107.45','ns_1@172.23.107.58',
      		 
      'ns_1@172.23.121.74','ns_1@172.23.121.78'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b5ce1e972169e0377441008a937cb49

      Snapshot: https://supportal.couchbase.com/snapshot/ff8cc1ad0a5b1fdc2c01fdd52cec4165::0

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-44725 [System Test]service_rebalance_failed,fts: rebalance_failed,inactivity_timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-45989 FTS: rollback not happening - FTS not catching up with KV

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-46154 FTS rebalance_out stuck for >1hr

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-46112
          # Subject Branch Project Status CR V
          152939,5 MB-46112: Log feed name when StartGocbcoreDCPFeed fails master cbgt Status: MERGED +2 +1
          152942,6 MB-46112: Notify manager only on agent error on feed initiation master cbgt Status: MERGED +2 +1

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty