Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-14802

[Windows] Error while doing rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Test Blocker
    • 4.0.0
    • 4.0.0
    • ns_server
    • Security Level: Public
    • Version: 4.0.0-2029 Enterprise Edition (build-2029)

    Description

      1. Install CB Server on 4 nodes
      2. Create bucket default and add 1000 documents on node 1
      3. Create 2 view and query them on node2
      4. Add node 2 to node 1

      Rebalance fails.
      Logs at - https://s3.amazonaws.com/bugdb/jira/MB-14802/logs.tar

      Here is output from test:
      2015-05-04 01:54:37 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.92:8091 to this cluster @172.23.105.82:8091
      2015-05-04 01:54:41 | INFO | MainProcess | test_thread | [rebalance_progress.test_progress_rebalance_in] Got empty progress
      2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.92%2Cns_1%40172.23.105.82
      2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
      2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      2015-05-04 01:54:53 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.105.82:
      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]

      {u'node': u'ns_1@172.23.105.92', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.92\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:40.642Z', u'module': u'ns_memcached', u'tstamp': 1430729680642, u'type': u'info'}

      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]

      {u'node': u'ns_1@172.23.105.92', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.92\'. Version: "4.0.0-2029-rel-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2015-05-04T01:54:40.470Z', u'module': u'menelaus_sup', u'tstamp': 1430729680470, u'type': u'info'}

      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]

      {u'node': u'ns_1@172.23.105.92', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.105.92\' for server shutdown', u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:35.579Z', u'module': u'ns_memcached', u'tstamp': 1430729675579, u'type': u'info'}

      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.82', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.18203.2>,\n error,{badrpc,nodedown,\n {gen_server,call,\n [

      {'janitor_agent-default',\n 'ns_1@172.23.105.92'}

      ,\n {if_rebalance,<0.18066.2>,\n {wait_index_updated,1020}},\n infinity]}}}}\n", u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:35.100Z', u'module': u'ns_orchestrator', u'tstamp': 1430729675100, u'type': u'info'}
      2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.82', u'code': 0, u'text': u"<0.18152.2> exited with {unexpected_exit,\n {'EXIT',<0.18203.2>,\n error,{badrpc,nodedown,\n {gen_server,call,\n [

      {'janitor_agent-default','ns_1@172.23.105.92'},\n {if_rebalance,<0.18066.2>,\n {wait_index_updated,1020}},\n infinity]}}}}", u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:35.085Z', u'module': u'ns_vbucket_mover', u'tstamp': 1430729675085, u'type': u'critical'}

      Errors from UI:
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.18203.2>,
      error,{badrpc,nodedown,
      {gen_server,call,
      [{'janitor_agent-default', 'ns_1@172.23.105.92'},
      {if_rebalance,<0.18066.2>,
      {wait_index_updated,1020}},
      infinity]}}}}
      ns_orchestrator002 ns_1@172.23.105.82 01:54:35 - Mon May 4, 2015
      <0.18152.2> exited with {unexpected_exit,
      {'EXIT',<0.18203.2>,
      error,{badrpc,nodedown,
      {gen_server,call,
      [{'janitor_agent-default','ns_1@172.23.105.92'}

      ,
      {if_rebalance,<0.18066.2>,
      {wait_index_updated,1020}},
      infinity]}}}} ns_vbucket_mover000 ns_1@172.23.105.82 01:54:35 - Mon May 4, 2015
      Bucket "default" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.82 01:54:34 - Mon May 4, 2015

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Aliaksey Artamonau Aliaksey Artamonau (Inactive)
            ritam.sharma Ritam Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty