Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8847

[system test][windows 2012] rebalance out failed with error "bulk_set_vbucket_state_failed, .."

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 2.2.0
    • 2.1.1
    • ns_server
    • Security Level: Public
    • windows 2012 standard 64-bit

    Description

      Environment:
      7 windows server 2012 standard 64-bit (each with 4 core cpu, 9GB RAM and 150 GB storage in single regular hard drive)
      1:172.23.105.184
      2:172.23.105.185
      3:172.23.105.186
      4:172.23.105.187
      5:172.23.105.188
      6:172.23.105.189
      7:172.23.105.190

      2 windows server 2012 with the same spec above use to add in and swap node test
      8:172.23.105.191
      9:172.23.105.192

      Cluster:
      Create 7 nodes cluster installed couchbase server 2.1.1-766
      Create 2 buckets with 1 replica: default and saslbucket (3.5GB per bucket)
      Load 25 million items size from 128 bytes to 512 bytes to each bucket.
      Continue to load until active resident reach 70% at each bucket.

      Procedure to test:
      Add node 191 to cluster ==> passed
      Swap node 184 with node 192 ==> passed
      Memcached crash on node 187 (will filed bug for this)
      Auto failover node 190 and rebalance ==> failed
      Tried rebalace few times later ==> falied

      Error for rebalance failed:

      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.30403.20>,
      {badmatch,
      [{'EXIT',
      {timeout,

      {gen_server,call, [<12838.5928.9>,had_backfill,30000]}}}]}}}
      ns_orchestrator002 ns_1@172.23.105.185 21:55:06 - Wed Aug 14, 2013
      <0.30396.20> exited with {unexpected_exit,
      {'EXIT',<0.30403.20>,
      {badmatch,
      [{'EXIT',
      {timeout,{gen_server,call, [<12838.5928.9>,had_backfill,30000]}

      }}]}}} ns_vbucket_mover000 ns_1@172.23.105.185 21:54:16 - Wed Aug 14, 2013

      Rebalance exited with reason {{bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.185',
      {'EXIT',
      {{{{unexpected_reason,killed},
      [

      {misc,executing_on_new_process,1}, {tap_replication_manager, change_vbucket_filter,4}, {tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^2/1-2-', 2}, {tap_replication_manager, do_set_incoming_replication_map,3}, {tap_replication_manager,handle_call,3}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]},
      {gen_server,call,
      ['tap_replication_manager-saslbucket', {change_vbucket_replication,929, 'ns_1@172.23.105.186'},
      infinity]}},
      {gen_server,call,
      [{'janitor_agent-saslbucket', 'ns_1@172.23.105.185'},
      {if_rebalance,<0.2067.21>,
      {update_vbucket_state,929,replica,
      undefined,'ns_1@172.23.105.186'}},
      infinity]}}}}]},
      [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,on_move_done,2},{gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}
      ns_orchestrator002 ns_1@172.23.105.185 23:52:08 - Wed Aug 14, 2013



      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.6651.24>,
      {badmatch,
      [{'EXIT',
      {timeout, {gen_server,call, [<12835.5816.18>,had_backfill, 30000]}}}]}}}
      ns_orchestrator002 ns_1@172.23.105.185 00:41:11 - Thu Aug 15, 2013
      <0.6642.24> exited with {unexpected_exit,
      {'EXIT',<0.6651.24>,
      {badmatch,
      [{'EXIT',
      {timeout, {gen_server,call, [<12835.5816.18>,had_backfill,30000]}}}]}}} ns_vbucket_mover000 ns_1@172.23.105.185 00:41:11 - Thu Aug 15, 2013



      Rebalance exited with reason {{bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.192',
      {'EXIT',
      {{{{unexpected_reason,killed},
      [{misc,executing_on_new_process,1}

      ,

      {tap_replication_manager, change_vbucket_filter,4}

      ,

      {tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^2/1-2-', 2}

      ,

      {tap_replication_manager, do_set_incoming_replication_map,3}

      ,

      {tap_replication_manager,handle_call,3}

      ,

      {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]},
      {gen_server,call,
      ['tap_replication_manager-saslbucket', {change_vbucket_replication,853, 'ns_1@172.23.105.188'},
      infinity]}},
      {gen_server,call,
      [{'janitor_agent-saslbucket', 'ns_1@172.23.105.192'},
      {if_rebalance,<0.16519.24>,
      {update_vbucket_state,853,replica,
      undefined,'ns_1@172.23.105.188'}},
      infinity]}}}}]},
      [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,on_move_done,2},{gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}
      ns_orchestrator002 ns_1@172.23.105.185 00:58:42 - Thu Aug 15, 2013

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thuan Thuan Nguyen
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty