Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5370

memcached core : #0 getVBucketId Rebalance failing with "Rebalance exited with reason {exited, {'EXIT',<0.10527.10>" .. {badmatch,{error,timeout} "

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 1.8.1
    • 1.8.1
    • ns_server
    • Security Level: Public
    • None
    • 14 Node cluster in DGM.
      Cluster can be found at 10.3.121.206
      1024 vbuckets, 1 Bucket

    Description

      Setup
      1. Setup a large cluster(14 nodes) with around 22M data cluster [10.3.121.206]
      2. Added 2 nodes on the cluster (224, 214) [Did a bunch of fail-overs, rebalance in/outs on this cluster prior to this]
      3. Rebalance-In the cluster

      The clusteris accessible in its current state @ http://10.3.121.228:8091/index.html#sec=log

      Output
      1. Rebalance fails as
      Rebalance exited with reason {exited,
      {'EXIT',<0.10527.10>,
      {badmatch,{error,timeout,
      [

      {mc_client_binary,stats_recv,4}, {mc_client_binary,stats,4}, {ns_memcached,do_handle_call,3}, {ns_memcached,handle_call,3}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]},
      {gen_server,call,
      [{'ns_memcached-default','ns_1@10.3.121.230'}, {stats,<<"tap">>},
      30000]}}}}
      ns_orchestrator002 ns_1@10.3.121.206 20:46:48 - Wed May 23, 2012
      <0.10526.10> exited with {exited,
      {'EXIT',<0.10527.10>,
      {badmatch,{error,timeout,
      [{mc_client_binary,stats_recv,4}

      ,

      {mc_client_binary,stats,4}

      ,

      {ns_memcached,do_handle_call,3}

      ,

      {ns_memcached,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]},
      {gen_server,call,
      [

      {'ns_memcached-default','ns_1@10.3.121.230'}

      ,

      {stats,<<"tap">>}

      ,
      30000]}}}}

      Attached the logs at https://s3.amazonaws.com/bugdb/jira/MB-5370/log_4.tar

      Attachments

        For Gerrit Dashboard: MB-5370
        # Subject Branch Project Status CR V

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty