Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19805

[windows] online upgrade from 3.1.1 to watson 2594 rebalance failed with error "dcp_wait_for_data_move_failed,"default",884..."

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • 3.1.1
    • couchbase-bucket
    • Windows 2012 R2 64 bit
    • Untriaged
    • Windows 64-bit
    • Yes

    Description

      Run test:
      newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_with_ops,initial_version=3.1.1-1807,sasl_buckets=1,standard_buckets=1,items=10000,nodes_init=2,nodes_in=1,nodes_out=1,ddocs-num=1,skip_init_check_cbserver=True,upgrade_version=4.5.0-2594,get-cbcollect-info=True

      {u'node': u'ns_1@172.23.107.119', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n                              {\'EXIT\',<0.1116.4>,\n                               {dcp_wait_for_data_move_failed,"default",884,\n                                \'ns_1@172.23.107.142\',\n                                [\'ns_1@172.23.107.119\',\'ns_1@172.23.105.91\'],\n                                {error,no_stats_for_this_vbucket}}}}\n', u'shortText': u'message', u'serverTime': u'2016-05-27T03:16:49.013Z', u'module': u'ns_orchestrator', u'tstamp': 1464344209013, u'type': u'info'}
      [2016-05-27 03:16:49,418] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u'<0.486.4> exited with {unexpected_exit,\n                          {\'EXIT\',<0.1116.4>,\n                              {dcp_wait_for_data_move_failed,"default",884,\n                                  \'ns_1@172.23.107.142\',\n                                  [\'ns_1@172.23.107.119\',\'ns_1@172.23.105.91\'],\n                                  {error,no_stats_for_this_vbucket}}}}', u'shortText': u'message', u'serverTime': u'2016-05-27T03:16:49.000Z', u'module': u'ns_vbucket_mover', u'tstamp': 1464344209000, u'type': u'critical'}
      [2016-05-27 03:16:49,418] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.105.91', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@172.23.105.91' disconnected: {badmatch,\n                                                                       {error,\n                                                                        closed}}", u'shortText': u'message', u'serverTime': u'2016-05-27T03:16:48.262Z', u'module': u'ns_memcached', u'tstamp': 1464344208262, u'type': u'info'}
      [2016-05-27 03:16:49,419] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.105.91', u'code': 0, u'text': u"Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 1. Restarting. Messages: Fri May 27 03:16:47.547422 Pacific Daylight Time 3: (bucket0) DCP (Producer) eq_dcpq:mapreduce_view: bucket0 _design/upgrade-test-view0 (prod/main) - (vb 50) stream created with start seqno 66 and end seqno 69\nFri May 27 03:16:47.547422 Pacific Daylight Time 3: (bucket0) DCP (Producer) eq_dcpq:mapreduce_view: bucket0 _design/upgrade-test-view0 (prod/main) - (vb 50) Sending disk snapshot with start seqno 66 and end seqno 70\nFri May 27 03:16:47.547422 Pacific Daylight Time 3: (bucket0) DCP (Producer) eq_dcpq:mapreduce_view: bucket0 _design/upgrade-test-view0 (prod/main) - (vb 50) Backfill complete, 0 items read from disk, last seqno read: 70\nFri May 27 03:16:47.548418 Pacific Daylight Time 3: (bucket0) DCP (Producer) eq_dcpq:mapreduce_view: bucket0 _design/upgrade-test-view0 (prod/main) - (vb 67) Backfill task (69 to 253403070514) finished. disk seqno 70 memory seqno 70\nFri May 27 03:16:47.548418 Pacific Daylight Time 3: (bucket0) DCP (Producer) eq_dcpq:mapreduce_view: bucket0 _design/upgrade-test-view0 (prod/main) - (vb 50) Stream closing, 3 items sent from disk, 0 items sent from memory, 70 was last seqno sent The stream ended due to all items being streamed is the reason", u'shortText': u'message', u'serverTime': u'2016-05-27T03:16:47.558Z', u'module': u'ns_log', u'tstamp': 1464344207558, u'type': u'info'}
      [2016-05-27 03:16:49,419] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-05-27T03:12:24.228Z', u'module': u'ns_vbucket_mover', u'tstamp': 1464343944228, u'type': u'info'}
      [2016-05-27 03:16:49,420] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.119\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-05-27T03:12:22.955Z', u'module': u'ns_memcached', u'tstamp': 1464343942955, u'type': u'info'}
      [2016-05-27 03:16:49,420] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.142', u'code': 0, u'text': u'Shutting down bucket "bucket0" on \'ns_1@172.23.107.142\' for deletion', u'shortText': u'message', u'serverTime': u'2016-05-27T03:12:22.443Z', u'module': u'ns_memcached', u'tstamp': 1464343942443, u'type': u'info'}
      .
      

      This test passed in build 4.5.0-2587

      Rerun again with build 4.5.0-2594. The test passed without any error.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty