Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8321

[Doc'd] windows32: Rebalance exited with reason {mover_failed,{badmatch,{error,eaddrinuse}}}

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
    • Environment:
      windows32, 2.0.2-807
    • Operating System:
      Windows 32-bit

      Description

      http://qa.hq.northscale.net/job/windows32_rebalance-kv/11/consoleFull

      ./testrunner -i /tmp/4-w-32.ini get-cbcollect-info=True -t rebalancetests.IncrementalRebalanceOut.test_load,replica=1,do-stop=True

      2013-05-19 06:24:15,085] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.179%2Cns_1%4010.3.2.185%2Cns_1%4010.3.2.178%2Cns_1%4010.3.2.183
      [2013-05-19 06:24:15,296] - [rest_client:929] INFO - rebalance operation started
      [2013-05-19 06:24:15,334] - [rest_client:1031] INFO - rebalance percentage : 0 %
      [2013-05-19 06:24:17,376] - [rest_client:1031] INFO - rebalance percentage : 0.0 %
      [2013-05-19 06:24:19,438] - [rest_client:1031] INFO - rebalance percentage : 0.703125 %
      [2013-05-19 06:24:21,473] - [rest_client:1031] INFO - rebalance percentage : 1.640625 %
      [2013-05-19 06:24:23,499] - [rest_client:1031] INFO - rebalance percentage : 2.6953125 %
      [2013-05-19 06:24:25,523] - [rest_client:1031] INFO - rebalance percentage : 3.6328125 %
      [2013-05-19 06:24:26,544] - [rest_client:1031] INFO - rebalance percentage : 4.21875 %
      [2013-05-19 06:24:27,572] - [rest_client:1031] INFO - rebalance percentage : 4.8046875 %
      [2013-05-19 06:24:28,601] - [rest_client:1031] INFO - rebalance percentage : 5.5078125 %
      [2013-05-19 06:24:30,639] - [rest_client:1031] INFO - rebalance percentage : 6.4453125 %
      [2013-05-19 06:24:31,663] - [rest_client:1014] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2013-05-19 06:24:31,663] - [rest_client:1015] INFO - Latest logs from UI:
      [2013-05-19 06:24:31,764] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.179\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969856238, u'type': u'info'}

      [2013-05-19 06:24:31,764] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.185' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969855691, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 3, u'text': u'Node ns_1@10.3.2.179 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969854738, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 4, u'text': u"Node 'ns_1@10.3.2.179' saw that node 'ns_1@10.3.2.183' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1368969854738, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 0, u'text': u"Port server moxi on node 'babysitter_of_ns_1@127.0.0.1' exited with status 0. Restarting. Messages: WARNING: curl error: transfer closed with outstanding read data remaining from: http://127.0.0.1:8091/pools/default/saslBucketsStreaming\nEOL on stdin. Exiting", u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1368969854628, u'type': u'info'}

      [2013-05-19 06:24:31,765] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.179', u'code': 1, u'text': u"Couchbase Server has started on web port 8091 on node 'ns_1@10.3.2.179'.", u'shortText': u'web start ok', u'module': u'menelaus_sup', u'tstamp': 1368969854597, u'type': u'info'}

      [2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 2, u'text': u'Rebalance exited with reason {mover_failed,{badmatch,

      {error,eaddrinuse}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368969854354, u'type': u'info'}
      [2013-05-19 06:24:31,766] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.178', u'code': 0, u'text': u'<0.26161.33> exited with {mover_failed,{badmatch,{error,eaddrinuse}

      }}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368969854339, u'type': u'critical'}
      [2013-05-19 06:24:31,767] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.185', u'code': 1, u'text': u'Bucket "bucket-0" loaded on node \'ns_1@10.3.2.185\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1368969850886, u'type': u'info'}

      [2013-05-19 06:24:31,767] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.2.185', u'code': 3, u'text': u'Node ns_1@10.3.2.185 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1368969850479, u'type': u'info'}

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          andreibaranouski Andrei Baranouski added a comment -

          closed according to above Karen's comments

          Show
          andreibaranouski Andrei Baranouski added a comment - closed according to above Karen's comments
          Hide
          siri Sriram Melkote added a comment - - edited

          Linking to MB-7902 which isn't quite resolved yet. If we close this bug independently of MB-7902, then it should be because setting MaxUserPorts resolves this issue independently

          Show
          siri Sriram Melkote added a comment - - edited Linking to MB-7902 which isn't quite resolved yet. If we close this bug independently of MB-7902 , then it should be because setting MaxUserPorts resolves this issue independently
          Hide
          andreibaranouski Andrei Baranouski added a comment -

          well, let's leave it open status

          but it should be noted that all runs/tests passed after applying the patch on the vms for http://qa.hq.northscale.net/job/windows32_rebalance-kv:

          Deepkaran Salooja added a comment - 27/May/13 3:33 PM
          Tested after applying fix:
          http://support.microsoft.com/kb/196271

          The test passes(build 2.0.2-812-rel):
          http://qa.hq.northscale.net/job/windows32_rebalance-kv/14/

          Show
          andreibaranouski Andrei Baranouski added a comment - well, let's leave it open status but it should be noted that all runs/tests passed after applying the patch on the vms for http://qa.hq.northscale.net/job/windows32_rebalance-kv: Deepkaran Salooja added a comment - 27/May/13 3:33 PM Tested after applying fix: http://support.microsoft.com/kb/196271 The test passes(build 2.0.2-812-rel): http://qa.hq.northscale.net/job/windows32_rebalance-kv/14/
          Hide
          kzeller kzeller added a comment - - edited

          [ADDED to Doc and RN]

          Sent to Tony:

          Hi Tony,

          There is a new prompt in the Windows installer so you do not run into port exhaustion. Please send me a screen shot of this new prompt.

          Thanks,

          Karen

          Show
          kzeller kzeller added a comment - - edited [ADDED to Doc and RN] Sent to Tony: Hi Tony, There is a new prompt in the Windows installer so you do not run into port exhaustion. Please send me a screen shot of this new prompt. Thanks, Karen
          Hide
          kzeller kzeller added a comment -

          Removing documentation label for 2.1.0. Save ticket for Eng/QA.

          Doc'd on 2.1.0 RN:

          The Windows installer for Windows 32-bit and 64-bit now prompts you to set the MaxUserPort registry setting. This will increase the number of ephemeral ports available to applications on Windows, as documented in Microsoft Knowledge Base Article 196271. The installer also warns you that a reboot is necessary for this change to take effect. If this registry key is not set, it may lead to port exhaustion leading to various problems, see as MB-8321. For installer instructions, see Section 2.2.3, “Microsoft Windows Installation”.

          Issues: MB-8321

          Main chapter update:

          http://www.couchbase.com/docs/couchbase-manual-2.1.0/couchbase-getting-started-install-win.html

          Show
          kzeller kzeller added a comment - Removing documentation label for 2.1.0. Save ticket for Eng/QA. Doc'd on 2.1.0 RN: The Windows installer for Windows 32-bit and 64-bit now prompts you to set the MaxUserPort registry setting. This will increase the number of ephemeral ports available to applications on Windows, as documented in Microsoft Knowledge Base Article 196271. The installer also warns you that a reboot is necessary for this change to take effect. If this registry key is not set, it may lead to port exhaustion leading to various problems, see as MB-8321 . For installer instructions, see Section 2.2.3, “Microsoft Windows Installation”. Issues: MB-8321 Main chapter update: http://www.couchbase.com/docs/couchbase-manual-2.1.0/couchbase-getting-started-install-win.html

            People

            • Assignee:
              andreibaranouski Andrei Baranouski
              Reporter:
              andreibaranouski Andrei Baranouski
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes