Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45893

Windows : Unable to add a new 7.0 node to a mixed mode cluster - pre_rebalance_janitor_run_failed

    XMLWordPrintable

Details

    Description

      Steps to repro :
      1. Setup a 2-node cluster running 6.6.2-9588 with kv+n1ql+index+fts services on both nodes
      2. Install the sample buckets and create FTS indexes
      3. Swap rebalance node2 with a spare node running 7.0.0-5016.
      4. The cluster will now be in mixed mode
      5. Add a new 7.0 node to the cluster.

      The rebalance fails with error : 

      Rebalance exited with reason {pre_rebalance_janitor_run_failed,
      		 
      "travel-sample",
      		 
      {error,wait_for_memcached_failed,
      		 
      ['ns_1@172.23.107.142']}}.
      		 
      Rebalance Operation Id = fda046d63e6f8a8b8014af2288bb470a

      6. Retried the rebalance. That too failed as ns_server on the existing 7.0 node 172.23.107.142 has crashed with the following error - 

      Service 'ns_server' exited with status 1. Restarting. Messages:
      		 
      working as port
      		 
      4092: Booted. Waiting for shutdown request
      		 
      working as port
      		 
      eheap_alloc: Cannot allocate 1573200 bytes of memory (of type "heap").
      		 
       
      		 
      Crash dump is being written to: erl_crash.dump...done
      		 
      [os_mon] win32 supervisor port (win32sysinfo): Erlang has closed

      Attachments

        1. screenshot-1.png
          screenshot-1.png
          32 kB
        2. screenshot-2.png
          screenshot-2.png
          37 kB
        3. screenshot-3.png
          screenshot-3.png
          31 kB
        4. screenshot-4.png
          screenshot-4.png
          48 kB

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-45935 XDCR - memory leak when pipeline restarts repeatedly

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-45895 [Windows] Online upgrade with swap rebalance failed with reason mover_crashed, wait_seqno_persisted_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-44823 XDCR hitting pools/nodes endpoint should be coordinated

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-45892 [Upgrade][XDCR][UI] Errors thrown while upgrading source cluster from 6.6.2 to 7.0.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty