Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8309

online_upgrade_swap_rebalance 1.8.1->2.0.2: Rebalance exited with reason {{change_filter_failed,{'EXIT',{{unexpected_reason,killed}

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.1.0
    • 2.1.0
    • ns_server
    • Security Level: Public
    • None
    • centos 32, 2.0.2-805-rel

    Description

      http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P1/4/consoleFull

      ./testrunner -i /tmp/centos-32-2.0-upgrade.ini upgrade_version=2.0.2-805-rel,get-cbcollect-info=True,GROUP=P1 -t newupgradetests.MultiNodesUpgradeTests.online_upgrade_swap_rebalance,initial_version=1.8.1-942-rel,standard_buckets=1,items=500000,max_verify=1000,GROUP=1_8;ONLINE;WINDOWS;P1

      steps:
      1)10.3.3.151, 10.3.3.152 with 1.8.1 build
      10.3.3.153 with 2.0.2-805-rel

      2)[2013-05-17 01:11:42,198] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.151&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.151%2Cns_1%4010.3.3.153
      ...
      [2013-05-17 01:26:15,908] - [task:340] INFO - rebalancing was completed with progress: 100.0% in 873.699863911 sec

      3)[2013-05-17 01:26:37,009] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.3.152&user=Administrator&knownNodes=ns_1%4010.3.3.152%2Cns_1%4010.3.3.153
      ...

      [2013-05-17 01:31:07,549] - [rest_client:1014] ERROR -

      {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

      - rebalance failed
      [2013-05-17 01:31:07,550] - [rest_client:1015] INFO - Latest logs from UI:
      [2013-05-17 01:31:07,601] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.3.153', u'code': 2, u'text': u"Rebalance exited with reason {{change_filter_failed,\n {'EXIT',\n {{unexpected_reason,killed},\n [

      {misc,executing_on_new_process,1}

      ,\n

      {ns_vbm_sup,local_change_vbucket_filter,4}

      ,\n

      {rpc,local_call,3}

      ,\n

      {ns_vbm_sup,change_vbucket_filter,4},\n {ns_vbm_sup,'-set_replicas/3-fun-2-',5},\n {lists,foldl,3},\n {ns_vbm_sup,set_replicas,3},\n {ns_vbm_sup,\n '-set_replicas_on_nodes/3-fun-1-',3}]}}},\n [{ns_vbm_sup,change_vbucket_filter,4}

      ,\n

      {ns_vbm_sup,'-set_replicas/3-fun-2-',5}

      ,\n

      {lists,foldl,3}

      ,\n

      {ns_vbm_sup,set_replicas,3}

      ,\n

      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',\n 3}

      ,\n

      {lists,foreach,2}

      ,\n

      {janitor_agent,\n do_bulk_set_vbucket_state_old_style,4}

      ,\n

      {ns_vbucket_mover,\n update_replication_post_move,3}

      ]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1368779444497.0, u'type': u'info'}
      [2013-05-17 01:31:07,602] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 43 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779418787.0, u'type': u'critical'}

      [2013-05-17 01:31:07,603] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358861.0, u'type': u'critical'}

      [2013-05-17 01:31:07,603] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 179 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779358787.0, u'type': u'critical'}

      [2013-05-17 01:31:07,604] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779299138.0, u'type': u'critical'}

      [2013-05-17 01:31:07,604] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 154 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779298786.0, u'type': u'critical'}

      [2013-05-17 01:31:07,605] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238798.0, u'type': u'critical'}

      [2013-05-17 01:31:07,605] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts (repeated 132 times)', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779238787.0, u'type': u'critical'}

      [2013-05-17 01:31:07,605] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Failed to get tap stats after 5 attempts', u'shortText': u'message', u'module': u'ebucketmigrator_srv', u'tstamp': 1368779184396.0, u'type': u'critical'}

      [2013-05-17 01:31:07,605] - [rest_client:1016] ERROR -

      {u'node': u'ns_1@10.3.3.153', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1368779184144.0, u'type': u'info'}

      Attachments

        For Gerrit Dashboard: MB-8309
        # Subject Branch Project Status CR V

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty