Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19073

[Windows] Rebalance of fts node fails with ""invalid character \'N\' looking for beginning of value" error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • cbft
    • None
    • Untriaged
    • Unknown

    Description

      Build
      4.5.0-2011

      Testcase
      test_29 in windows fts rebalance suite

      ./testrunner -i /tmp/testexec.27410.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,items=10000,query_max_matches=10000 -t fts.moving_topology_fts.MovingTopFTS.rebalance_in_between_indexing_and_querying,items=30000,cluster=D,F,D+F,sasl_buckets=1,GROUP=P0

      Rebalance fails while trying to add 172.23.107.37 to cluster 172.23.106.181.

      [2016-04-04 17:26:23,779] - [fts_base:1621] INFO - Starting rebalance-in nodes:[ip:172.23.107.37 port:8091 ssh_username:Administrator] at C1 cluster 172.23.106.181
      [2016-04-04 17:26:24,535] - [task:404] INFO - adding node 172.23.107.37:8091 to cluster
      [2016-04-04 17:26:24,536] - [rest_client:988] INFO - adding remote node @172.23.107.37:8091 to this cluster @172.23.106.181:8091
      [2016-04-04 17:26:29,008] - [rest_client:1206] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.146%2Cns_1%40172.23.106.182%2Cns_1%40172.23.106.181%2Cns_1%40172.23.107.37
      [2016-04-04 17:26:29,013] - [rest_client:1210] INFO - rebalance operation started
      [2016-04-04 17:26:29,018] - [rest_client:1338] INFO - rebalance percentage : 0.00 %
      [2016-04-04 17:26:39,038] - [rest_client:1338] INFO - rebalance percentage : 10.82 %
      [2016-04-04 17:26:49,057] - [rest_client:1338] INFO - rebalance percentage : 23.86 %
      [2016-04-04 17:26:59,127] - [rest_client:1338] INFO - rebalance percentage : 34.38 %
      [2016-04-04 17:27:09,145] - [rest_client:1338] INFO - rebalance percentage : 46.67 %
      [2016-04-04 17:27:19,164] - [rest_client:1321] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
      [2016-04-04 17:27:19,249] - [rest_client:2443] INFO - Latest logs from UI on 172.23.106.181:
      [2016-04-04 17:27:19,249] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 2, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n                              {rebalance_failed,\n                               {service_error,\n                                <<"invalid character \'N\' looking for beginning of value">>}}}\n', u'shortText': u'message', u'serverTime': u'2016-04-04T17:27:12.926Z', u'module': u'ns_orchestrator', u'tstamp': 1459816032926, u'type': u'info'}
      [2016-04-04 17:27:19,249] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:50.928Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459816010928, u'type': u'info'}
      [2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.37\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:50.091Z', u'module': u'ns_memcached', u'tstamp': 1459816010091, u'type': u'info'}
      [2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:49.769Z', u'module': u'ns_rebalancer', u'tstamp': 1459816009769, u'type': u'info'}
      [2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Bucket "sasl_bucket_1" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:30.158Z', u'module': u'ns_vbucket_mover', u'tstamp': 1459815990158, u'type': u'info'}
      [2016-04-04 17:27:19,250] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 0, u'text': u'Bucket "sasl_bucket_1" loaded on node \'ns_1@172.23.107.37\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:29.859Z', u'module': u'ns_memcached', u'tstamp': 1459815989859, u'type': u'info'}
      [2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 3, u'text': u'Node ns_1@172.23.107.37 joined cluster', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:29.207Z', u'module': u'ns_cluster', u'tstamp': 1459815989207, u'type': u'info'}
      [2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.107.37', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.37\'. Version: "4.5.0-2011-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2016-04-04T17:26:29.147Z', u'module': u'menelaus_sup', u'tstamp': 1459815989147, u'type': u'info'}
      [2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 0, u'text': u'Started rebalancing bucket sasl_bucket_1', u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:28.997Z', u'module': u'ns_rebalancer', u'tstamp': 1459815988997, u'type': u'info'}
      [2016-04-04 17:27:19,251] - [rest_client:2444] ERROR - {u'node': u'ns_1@172.23.106.181', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.146','ns_1@172.23.106.182',\n                                 'ns_1@172.23.106.181','ns_1@172.23.107.37'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-04-04T17:26:28.955Z', u'module': u'ns_orchestrator', u'tstamp': 1459815988955, u'type': u'info'}
      ERROR
      

      Attaching cbcollect info.

      Attachments

        1. 172.23.106.181-20160404-1727-diag.zip
          34.40 MB
        2. 172.23.106.182-20160404-1730-diag.zip
          17.73 MB
        3. 172.23.107.146-20160404-1732-diag.zip
          13.02 MB
        4. 172.23.107.37-20160404-1735-diag.zip
          18.02 MB
        5. collectinfo-2016-04-11T193216-ns_1@172.23.123.155.zip
          22.90 MB
        6. collectinfo-2016-04-12T015317-ns_1@172.23.123.155.zip
          40.63 MB
        7. fts.log
          610 kB
        8. fts.log
          1.29 MB
        9. fts.log
          2.10 MB

        Issue Links

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty