Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11724

3.0 cluster is briefly down after remove last 2.0 nodes out of cluster in online upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • ns_server
    • Security Level: Public
    • None
    • ubuntu 12.04 64-bit

    Description

      Install cb server 2.1.1-764 on 2 nodes (148 and 149)
      Create cluster of 2 nodes
      Create default bucket and load 5000 items to default bucket.
      Install cb server 3.0.0-960 on 2 nodes (150 and 151)
      Add node 150 and 151 into 2.1.1 cluster.
      Rebalance. Passed
      Remove 2 2.1.1 nodes (148 and 149) out of cluster
      Rebalance. Passed.
      Right after rebalance is finished. 3.0 cluster with 2 nodes (150 and 151) is briefly down about 30 to 45 seconds
      Then 3.0 cluster is up again after that.

      Output from test run
      2014-07-14 14:05:22,473 - root - INFO - rebalance params : password=password&ejectedNodes=ns_1%40192.168.171.148%2Cns_1%40192.168.171.149&user=Administrator&knownNodes=ns_1%40192.168.171.149%2Cns_1%40192.168.171.151%2Cns_1%40192.168.171.148%2Cns_1%40192.168.171.150
      2014-07-14 14:05:22,481 - root - INFO - rebalance operation started
      2014-07-14 14:05:22,489 - root - INFO - rebalance percentage : 0 %
      2014-07-14 14:05:32,499 - root - INFO - rebalance percentage : 100.0 %
      2014-07-14 14:05:41,120 - root - ERROR - socket error while connecting to http://192.168.171.148:8091/pools error [Errno 61] Connection refused
      2014-07-14 14:05:42,122 - root - ERROR - socket error while connecting to http://192.168.171.148:8091/pools error [Errno 61] Connection refused
      2014-07-14 14:05:43,136 - root - INFO - rebalancing was completed with progress: 100.0% in 20.654556036 sec
      2014-07-14 14:05:43,137 - root - INFO - sleep for 15 secs. ...
      2014-07-14 14:05:58,148 - root - INFO - existing buckets : [u'default']
      2014-07-14 14:05:58,148 - root - INFO - found bucket default
      2014-07-14 14:05:59,158 - root - INFO - creating direct client 192.168.171.150:11210 default
      2014-07-14 14:05:59,187 - root - INFO - Saw ep_queue_size 0 == 0 expected on '192.168.171.150:8091',default bucket
      2014-07-14 14:05:59,190 - root - ERROR - http://192.168.171.151:8091/nodes/self error 401 reason: status: 401, content:
      http://192.168.171.151:8091/nodes/self:
      2014-07-14 14:05:59,193 - root - ERROR - http://192.168.171.151:8091/nodes/self error 401 reason: status: 401, content:
      2014-07-14 14:05:59,196 - root - ERROR - http://192.168.171.151:8091/pools/default error 401 reason: status: 401, content:
      2014-07-14 14:05:59,196 - root - INFO - sleep 10 seconds and retry
      2014-07-14 14:06:09,200 - root - ERROR - http://192.168.171.151:8091/pools/default error 401 reason: status: 401, content:
      2014-07-14 14:06:09,200 - root - INFO - sleep 10 seconds and retry
      2014-07-14 14:06:19,205 - root - INFO - creating direct client 192.168.171.151:11210 default
      2014-07-14 14:06:19,230 - root - INFO - Saw ep_queue_size 0 == 0 expected on '192.168.171.151:8091',default bucket
      2014-07-14 14:06:19,277 - root - INFO - creating direct client 192.168.171.151:11210 default

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            artem Artem Stemkovski
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty