Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7632

[windows]Rebalance exited with reason {bulk_set_vbucket_state_failed,

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 3.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
    • Environment:

      Description

      -t rebalancetests.IncrementalRebalanceOut.test_load,do-stop:True,,num_nodes:6,replica:2

      rebalance in 1->6 nodes

      2013-01-29 12:22:35.905 ns_vbucket_mover:0:critical:message(ns_1@10.1.3.82) - <0.15039.67> exited with {bulk_set_vbucket_state_failed,
      [{'ns_1@10.1.3.85',
      {'EXIT',
      {{{{unexpected_reason,
      badmatch,{error,closed,
      [

      {mc_binary,quick_stats_recv,3},
      {mc_binary,quick_stats_loop,5},
      {mc_binary,quick_stats,5},
      {mc_client_binary, get_zero_open_checkpoint_vbuckets,3},
      {ebucketmigrator_srv,handle_call,3},
      {gen_server,handle_msg,5},
      {proc_lib,init_p_do_apply,3}]}},
      [{misc,executing_on_new_process,1},
      {tap_replication_manager, change_vbucket_filter,4},
      {tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^5/1-5-', 2},
      {tap_replication_manager, do_set_incoming_replication_map,3},
      {tap_replication_manager,handle_call,3},
      {gen_server,handle_msg,5},
      {proc_lib,init_p_do_apply,3}]},
      {gen_server,call,
      ['tap_replication_manager-bucket-0',
      {change_vbucket_replication,319,undefined},
      infinity]}},
      {gen_server,call,
      [{'janitor_agent-bucket-0','ns_1@10.1.3.85'},
      {if_rebalance,<0.7485.67>,
      {update_vbucket_state,319,replica,passive,
      undefined}},
      infinity]}}}}]}
      2013-01-29 12:22:38.177 ns_memcached:1:info:message(ns_1@10.1.3.84) - Bucket "bucket-0" loaded on node 'ns_1@10.1.3.84' in 0 seconds.
      2013-01-29 12:22:40.671 ns_orchestrator:2:info:message(ns_1@10.1.3.82) - Rebalance exited with reason {bulk_set_vbucket_state_failed,
      [{'ns_1@10.1.3.85',
      {'EXIT',
      {{{{unexpected_reason,
      badmatch,{error,closed,
      [{mc_binary,quick_stats_recv,3}

      ,

      {mc_binary,quick_stats_loop,5}

      ,

      {mc_binary,quick_stats,5}

      ,

      {mc_client_binary, get_zero_open_checkpoint_vbuckets,3}

      ,

      {ebucketmigrator_srv,handle_call,3}

      ,

      {gen_server,handle_msg,5},
      {proc_lib,init_p_do_apply,3}]}},
      [{misc,executing_on_new_process,1},
      {tap_replication_manager, change_vbucket_filter,4},
      {tap_replication_manager, '-do_set_incoming_replication_map/3-lc$^5/1-5-', 2},
      {tap_replication_manager, do_set_incoming_replication_map,3},
      {tap_replication_manager,handle_call,3},
      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]},
      {gen_server,call,
      ['tap_replication_manager-bucket-0',

      {change_vbucket_replication,319, undefined}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-bucket-0', 'ns_1@10.1.3.85'}

      ,
      {if_rebalance,<0.7485.67>,
      {update_vbucket_state,319,replica,
      passive,undefined}},
      infinity]}}}}]}

      attaching cbcollect

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        iryna iryna added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.82.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.83.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.84.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.85.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.86.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/8b18cad6/10.1.3.87.zip
        Hide
        iryna iryna added a comment -

        tested manually, not reproduced

        Show
        iryna iryna added a comment - tested manually, not reproduced
        Show
        iryna iryna added a comment - - edited reproduced with build 2.0.1-160-rel logs: https://s3.amazonaws.com/bugdb/jira/MB-7632/4e64d520/10.1.3.84-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/4e64d520/10.1.3.82-diag.zip https://s3.amazonaws.com/bugdb/jira/MB-7632/4e64d520/10.1.3.85-diag.zip
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Another case of memcached simply dropping connection.

        We were in the beginning of new-style vbucket filter change.

        We just started by sending stats request to source node to find out which vbuckets have 0 open checkpoint. This is done to avoid replication from vbuckets that are backfilled in (relevant only for chain replication, i.e. replication from replica). And instead of getting result back we've got closed connection.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Another case of memcached simply dropping connection. We were in the beginning of new-style vbucket filter change. We just started by sending stats request to source node to find out which vbuckets have 0 open checkpoint. This is done to avoid replication from vbuckets that are backfilled in (relevant only for chain replication, i.e. replication from replica). And instead of getting result back we've got closed connection.
        Hide
        jin Jin Lim added a comment -

        Hi Iryna - I am going to assign this back to you. Please rerun the test after Windows 2.0.1 is released. Thanks.

        Show
        jin Jin Lim added a comment - Hi Iryna - I am going to assign this back to you. Please rerun the test after Windows 2.0.1 is released. Thanks.
        Hide
        mikew Mike Wiederhold added a comment -

        Also see MB-7060

        Show
        mikew Mike Wiederhold added a comment - Also see MB-7060
        Hide
        iryna iryna added a comment -

        build 2.0.2-761-rel not reproduced anymore

        Show
        iryna iryna added a comment - build 2.0.2-761-rel not reproduced anymore
        Hide
        iryna iryna added a comment -

        verified 2.0.2-761-rel

        Show
        iryna iryna added a comment - verified 2.0.2-761-rel

          People

          • Assignee:
            iryna iryna
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes