Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7370

[system test] memcached crashed when add node back in online upgrade from 1.8.1 to 2.0.0

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.1
    • Component/s: ns_server
    • Security Level: Public
    • Environment:
      windows 2008 R2 64bit

      Description

      Install couchbase server 1.8.1 on 4 nodes cluster with default installation path and custom data path.
      10.3.2.10
      10.3.2.11
      10.3.2.12
      10.3.2.16
      10.3.2.75

      Create 3 buckets, one default, one sasl and one dedicated port bucket.
      Load 20K items to each bucket.
      Maintain load about 3K at each bucket.

      Install couchbase server 2.0.0-1974 on node 10.3.2.76 with default installation path and custom data path.
      Swap rebalance (add node 76 and remove node 75). Rebalance done.

      Install couchbase server 2.0.0-1974 on node 10.3.2.77 with default installation path and custom data path.
      Swap rebalance (add node 77 and remove node 16). Rebalance done.

      Uninstall couchbase server 1.8.1 on node 16 and 75
      Install couchbase server 2.0.0-1974 on node 16 and 75 with custom installation path (c:/one) and custom data path (c:/data)
      Swap rebalance (add node 16 and 75 to cluster, remove node 11 and 12)
      Rebalance failed in few minutes

      I see a lot of error in log page with error

      Port server memcached on node 'ns_1@10.3.2.16' exited with status 1. Restarting. Messages: Wed Dec 05 16:16:29.949804 Pacific Standard Time 3: Error initializing sasl.

      Filed bug MB-7369

      Then clean install node 16 and node 75 with default installation path (c:/Program Files/Couchbase/Server/)
      Add node 16 and 75 back to cluster
      10.3.2.10
      10.3.2.11
      10.3.2.12
      10.3.2.76
      10.3.2.77

      Remove node 11 and 12 to do swap rebalance.
      Rebalance failed. Check memcached on node 16 and 75, I see memcached crashed continuously
      I see the bucket with dedicated port (port 11207) is in pending state as in capture

      The cluster is in failed state now.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        if i understand correctly here , path was changed when node was failed over before being added back to the cluster ?

        if thats the case this is not a normal scenario for customers

        Show
        farshid Farshid Ghods (Inactive) added a comment - if i understand correctly here , path was changed when node was failed over before being added back to the cluster ? if thats the case this is not a normal scenario for customers
        Hide
        thuan Thuan Nguyen added a comment -

        It is not failover operation. It is swap rebalane upgrade. Customer may decide to change the path of all nodes at time of upgrade because they have to swap all nodes in cluster.

        Show
        thuan Thuan Nguyen added a comment - It is not failover operation. It is swap rebalane upgrade. Customer may decide to change the path of all nodes at time of upgrade because they have to swap all nodes in cluster.
        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - http://review.couchbase.org/23124
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        per bug scrub
        please upload the fix to 2.0.1 branch

        Show
        farshid Farshid Ghods (Inactive) added a comment - per bug scrub please upload the fix to 2.0.1 branch
        Show
        farshid Farshid Ghods (Inactive) added a comment - merged http://review.couchbase.org/#/c/23376/

          People

          • Assignee:
            Aliaksey Artamonau Aliaksey Artamonau
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes