Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15033

Error while adding nodes and rebalance - swap rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 4.0.0
    • 4.0.0
    • ns_server
    • Security Level: Public
    • 4.0.0-2157 DEV Edition (build-2157)

    Description

      1. Install CB Server on 6 nodes
      2. Create 4 buckets
      3. Add 2 nodes to the cluster and rebalance
      4. Start loading data to the cluster
      5. Remove 2 nodes but do not rebalance
      6. Add 2 nodes to the cluster
      7. Start rebalance

      Error:
      [2015-05-17 23:22:39,435] - [rest_client:731] ERROR - http://10.3.5.115:8091/controller/addNode error 400 reason: unknown ["Join completion call failed. Got HTTP status 500 from REST call post to http://10.3.5.118:8091/completeJoin. Body was: \"\\\"Unexpected server error, request logged.\\\"\""]
      [2015-05-17 23:22:39,477] - [rest_client:2186] INFO - Latest logs from UI on 10.3.5.115:
      [2015-05-17 23:22:39,478] - [rest_client:2187] ERROR -

      {u'node': u'ns_1@10.3.5.115', u'code': 5, u'text': u'Failed to add node 10.3.5.118:8091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to http://10.3.5.118:8091/completeJoin. Body was: "[\\"Unexpected server error, request logged.\\"]"', u'shortText': u'message', u'serverTime': u'2015-05-17T23:22:39.466Z', u'module': u'ns_cluster', u'tstamp': 1431930159466, u'type': u'info'}

      [2015-05-17 23:22:39,478] - [rest_client:2187] ERROR - {u'node': u'ns_1@10.3.5.115', u'code': 0, u'text': u'Add transaction of \'ns_1@10.3.5.118\' failed because of {error,complete_join,\n <<"Join completion call failed. Got HTTP status 500 from REST call post to http://10.3.5.118:8091/completeJoin. Body was: \\"\\\\\\"Unexpected server error, request logged.\\\\\\"\\"">>,\n {error,rest_error,\n <<"Got HTTP status 500 from REST call post to http://10.3.5.118:8091/completeJoin. Body was: \\"\\\\\\"Unexpected server error, request logged.\\\\\\"\\"">>,\n

      {bad_status,500,\n <<"[\\"Unexpected server error, request logged.\\"]">>}

      }}', u'shortText': u'message', u'serverTime': u'2015-05-17T23:22:39.462Z', u'module': u'ns_cluster', u'tstamp': 1431930159462, u'type': u'critical'}
      [2015-05-17 23:22:39,479] - [rest_client:2187] ERROR -

      {u'node': u'ns_1@10.3.5.115', u'code': 0, u'text': u"Started node add transaction by adding node 'ns_1@10.3.5.118' to nodes_wanted (group: undefined)", u'shortText': u'message', u'serverTime': u'2015-05-17T23:22:39.441Z', u'module': u'ns_cluster', u'tstamp': 1431930159441, u'type': u'info'}

      TC:
      ./testrunner i centos_x64-rebalance_out.ini get-cbcollect-info=True,get-logs=True,stop-on-failure=False,get-coredumps=True,force_kill_memached=False,verify_unacked_bytes=True,std_vbucker_dist=8,chk_max_items=1000,chk_period=30 -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=1,num-buckets=4,num-swap=2,swap-orchestrator=True,percentage_progress=30,GROUP=P0

      TC trace uploaded to defect - MB-15033.rtf

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              artem Artem Stemkovski
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty