Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8321

[Doc'd] windows32: Rebalance exited with reason {mover_failed,{badmatch,{error,eaddrinuse}}}

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
    • Environment:
      windows32, 2.0.2-807
    • Operating System:
      Windows 32-bit

      Description

      http://qa.hq.northscale.net/job/windows32_rebalance-kv/11/consoleFull

      ./testrunner -i /tmp/4-w-32.ini get-cbcollect-info=True -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,do-stop=True

      2013-05-19 06:24:15,085] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.179%2Cns_1%4010.3.2.185%2Cns_1%4010.3.2.178%2Cns_1%4010.3.2.183
      [2013-05-19 06:24:15,296] - [rest_client:929] INFO - rebalance operation started
      [2013-05-19 06:24:15,334] - [rest_client:1031] INFO - rebalance percentage : 0 %
      [2013-05-19 06:24:17,376] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
      [2013-05-19 06:24:19,438] - [rest_client:1031] INFO - rebalance percentage : 0.703125 %
      [2013-05-19 06:24:21,473] - [rest_client:1031] INFO - rebalance percentage : 1.640625 %
      [2013-05-19 06:24:23,499] - [rest_client:1031] INFO - rebalance percentage : 2.6953125 %
      [2013-05-19 06:24:25,523] - [rest_client:1031] INFO - rebalance percentage : 3.6328125 %
      [2013-05-19 06:24:26,544] - [rest_client:1031] INFO - rebalance percentage : 4.21875 %
      [2013-05-19 06:24:27,572] - [rest_client:1031] INFO - rebalance percentage : 4.8046875 %
      [2013-05-19 06:24:28,601] - [rest_client:1031] INFO - rebalance percentage : 5.5078125 %
      [2013-05-19 06:24:30,639] - [rest_client:1031] INFO - rebalance percentage : 6.4453125 %
      [2013-05-19 06:24:31,663] - [rest_client:1014] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2013-05-19 06:24:31,663] - [rest_client:1015] INFO - Latest logs from UI:
      [2013-05-19 06:24:31,764] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.179\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969856238, u'type': u'info'}

      [2013-05-19 06:24:31,764] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.185' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969855691, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 3, u'text': u'Node ns_1@10.3.2.179 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969854738, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.183' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969854738, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 0, u'text': u"Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming\nEOL on stdin. Exiting", u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368969854628, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u"Couchbase Server has started on web port 8091 on node 'ns_1@10.3.2.179'.", u'shortText': u'web start ok', u'module': u'menelaus_sup', u'tstamp': 1368969854597, u'type': u'info'}

      [2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 2, u'text': u'Rebalance exited with reason {mover_failed,{badmatch,

      {error,eaddrinuse}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368969854354, u'type': u'info'}
      [2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 0, u'text': u'<0.26161.33> exited with {mover_failed,{badmatch,{error,eaddrinuse}

      }}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368969854339, u'type': u'critical'}
      [2013-05-19 06:24:31,767] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.185', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.185\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969850886, u'type': u'info'}

      [2013-05-19 06:24:31,767] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.185', u'code': 3, u'text': u'Node ns_1@10.3.2.185 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969850479, u'type': u'info'}

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            • Assignee:
              andreibaranouski Andrei Baranouski
              Reporter:
              andreibaranouski Andrei Baranouski
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes