Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5625

Rebalance can fail with various error if user changes bucket password before rebalance process

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.0-beta
    • 1.8.1
    • ns_server
    • Security Level: Public
    • Large Cluster - Centos, - 16 node cluster
      Build 181-916rel
      3 buckets - bucket1(3G), bucket2(2.8G), bucket3(200M)

    Description

      Setup:
      1.Setup a 18 node cluster. Enable Auto-failover
      2.Load data on all 3 buckets [around 50M, 22M, 500k] items.
      3. Continue loading data..
      4. Remove orchestrator node [105] Add new node [126]
      5. Issue rebalance on this cluster. Rebalance failed with "replicator died" - filed bug 5343
      6. Re-issue rebalance . Rebalance fails with " change_filter_failed"

      Output:
      Reissue rebalance fails with "change_filter_failed"
      Node 105 is auto-failed over

      Rebalance exited with reason {{change_filter_failed,
      {'EXIT',
      {{badmatch,
      {failed,
      {error,
      {badmatch,
      {memcached_error,auth_error,
      <<"Auth failure">>}},
      [

      {ebucketmigrator_srv,connect,4}

      ,

      {ebucketmigrator_srv,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}}},
      [

      {ns_vbm_sup, '-local_change_vbucket_filter/4-fun-2-',6}

      ,

      {misc,'-executing_on_new_process/1-fun-0-', 3}

      ]}}},
      [

      {ns_vbm_sup,change_vbucket_filter,4}

      ,

      {ns_vbm_sup,'-set_replicas/3-fun-2-',5}

      ,

      {lists,foldl,3}

      ,

      {ns_vbm_sup,set_replicas,3}

      ,

      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-', 3}

      ,

      {lists,foreach,2}

      ,

      {ns_vbm_sup,apply_changes,2}

      ,

      {ns_vbucket_mover,sync_replicas,0}

      ]}

      Logs at - https://s3.amazonaws.com/bugdb/jira/bug4-rebalance-181/bug5.tar

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty