Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45893

Windows : Unable to add a new 7.0 node to a mixed mode cluster - pre_rebalance_janitor_run_failed

    XMLWordPrintable

Details

    Description

      Steps to repro :
      1. Setup a 2-node cluster running 6.6.2-9588 with kv+n1ql+index+fts services on both nodes
      2. Install the sample buckets and create FTS indexes
      3. Swap rebalance node2 with a spare node running 7.0.0-5016.
      4. The cluster will now be in mixed mode
      5. Add a new 7.0 node to the cluster.

      The rebalance fails with error :

      Rebalance exited with reason {pre_rebalance_janitor_run_failed,
      "travel-sample",
      {error,wait_for_memcached_failed,
      ['ns_1@172.23.107.142']}}.
      Rebalance Operation Id = fda046d63e6f8a8b8014af2288bb470a
      

      6. Retried the rebalance. That too failed as ns_server on the existing 7.0 node 172.23.107.142 has crashed with the following error -

      Service 'ns_server' exited with status 1. Restarting. Messages:
      working as port
      4092: Booted. Waiting for shutdown request
      working as port
      eheap_alloc: Cannot allocate 1573200 bytes of memory (of type "heap").
       
      Crash dump is being written to: erl_crash.dump...done
      [os_mon] win32 supervisor port (win32sysinfo): Erlang has closed
      

      Attachments

        1. screenshot-1.png
          screenshot-1.png
          32 kB
        2. screenshot-2.png
          screenshot-2.png
          37 kB
        3. screenshot-3.png
          screenshot-3.png
          31 kB
        4. screenshot-4.png
          screenshot-4.png
          48 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            mihir.kamdar Mihir Kamdar (Inactive) created issue -
            bharath.gp Bharath G P made changes -
            Field Original Value New Value
            Link to Log File, atop/blg, CBCollectInfo, Core dump https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.126.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.129.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.142.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.126.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.129.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/logs/mixed_mode_cluster_add_new_node/collectinfo-2021-04-23T230138-ns_1%40172.23.107.142.zip

            Logs from Bharath
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.104.247.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.105.1.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.105.137.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.105.183.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.106.116.zip : Upgrade node (7.0.0-5017)
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.126.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.127.zip : Node being swapped out
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.129.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.131.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.142.zip
            https://cb-jira.s3.us-east-2.amazonaws.com/MB-45893/collectinfo-2021-04-26T111620-ns_1%40172.23.107.146.zip
            ritam.sharma Ritam Sharma made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            ritam.sharma Ritam Sharma made changes -
            Labels upgrade windows affects-cc-testing upgrade windows
            dfinlay Dave Finlay made changes -
            Attachment screenshot-1.png [ 137432 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-2.png [ 137433 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-3.png [ 137434 ]
            dfinlay Dave Finlay made changes -
            Attachment screenshot-4.png [ 137456 ]
            dfinlay Dave Finlay made changes -
            Assignee Dave Finlay [ dfinlay ] Neil Huang [ neil.huang ]
            dfinlay Dave Finlay made changes -
            Component/s XDCR [ 10136 ]
            Component/s ns_server [ 10019 ]
            neil.huang Neil Huang made changes -
            Assignee Neil Huang [ neil.huang ] Mihir Kamdar [ mihir.kamdar ]
            neil.huang Neil Huang made changes -
            Link This issue relates to MB-44823 [ MB-44823 ]
            neil.huang Neil Huang made changes -
            Link This issue relates to MB-45892 [ MB-45892 ]
            neil.huang Neil Huang made changes -
            Link This issue is caused by MB-45935 [ MB-45935 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            neil.huang Neil Huang made changes -
            Link This issue is duplicated by MB-45895 [ MB-45895 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty