Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5625

Rebalance can fail with various error if user changes bucket password before rebalance process

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.1
    • Fix Version/s: 2.0-beta
    • Component/s: ns_server
    • Security Level: Public
    • Environment:
      Large Cluster - Centos, - 16 node cluster
      Build 181-916rel
      3 buckets - bucket1(3G), bucket2(2.8G), bucket3(200M)

      Description

      Setup:
      1.Setup a 18 node cluster. Enable Auto-failover
      2.Load data on all 3 buckets [around 50M, 22M, 500k] items.
      3. Continue loading data..
      4. Remove orchestrator node [105] Add new node [126]
      5. Issue rebalance on this cluster. Rebalance failed with "replicator died" - filed bug 5343
      6. Re-issue rebalance . Rebalance fails with " change_filter_failed"

      Output:
      Reissue rebalance fails with "change_filter_failed"
      Node 105 is auto-failed over

      Rebalance exited with reason {{change_filter_failed,
      {'EXIT',
      {{badmatch,
      {failed,
      {error,
      {badmatch,
      {memcached_error,auth_error,
      <<"Auth failure">>}},
      [

      {ebucketmigrator_srv,connect,4}

      ,

      {ebucketmigrator_srv,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}}},
      [

      {ns_vbm_sup, '-local_change_vbucket_filter/4-fun-2-',6}

      ,

      {misc,'-executing_on_new_process/1-fun-0-', 3}

      ]}}},
      [

      {ns_vbm_sup,change_vbucket_filter,4}

      ,

      {ns_vbm_sup,'-set_replicas/3-fun-2-',5}

      ,

      {lists,foldl,3}

      ,

      {ns_vbm_sup,set_replicas,3}

      ,

      {ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-', 3}

      ,

      {lists,foreach,2}

      ,

      {ns_vbm_sup,apply_changes,2}

      ,

      {ns_vbucket_mover,sync_replicas,0}

      ]}

      Logs at - https://s3.amazonaws.com/bugdb/jira/bug4-rebalance-181/bug5.tar

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        changing bucket password impacts all ongoing TAP and newer connections between memcached on all nodes and changing the bucket password without re-establishing all those connections or establishing new connections without using the new password will cause those issues

        Show
        farshid Farshid Ghods (Inactive) added a comment - changing bucket password impacts all ongoing TAP and newer connections between memcached on all nodes and changing the bucket password without re-establishing all those connections or establishing new connections without using the new password will cause those issues
        Hide
        perry Perry Krug added a comment -

        Just had a customer run into this...is there any workaround before the fix? Do we know if simply retrying the rebalance will work?

        Does this mean that replication streams are stopped as well?

        What about resetting the replica chains?

        Show
        perry Perry Krug added a comment - Just had a customer run into this...is there any workaround before the fix? Do we know if simply retrying the rebalance will work? Does this mean that replication streams are stopped as well? What about resetting the replica chains?
        Hide
        peter peter added a comment -

        Alk, is this still an issue in 2.0 and if not, what is the workaround for 1.8.1? Change the password back?

        Show
        peter peter added a comment - Alk, is this still an issue in 2.0 and if not, what is the workaround for 1.8.1? Change the password back?
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Doesn't apply to 2.0 in 2.0 compat mode. Will still be a problem in 1.8.1 compat mode.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Doesn't apply to 2.0 in 2.0 compat mode. Will still be a problem in 1.8.1 compat mode.

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            ketaki Ketaki Gangal
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes