Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5513

Rebalance failure due to replicator_died with "memcached_error,auth_error,<<"Auth failure">>" because node was marked as ready before isasl.pw was read

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.8.1-release-candidate
    • Fix Version/s: 1.8.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Ubuntu 64 bit
      181-901-rel

      Description

      Decided to file a new bug, but the old one is here:-
      http://www.couchbase.com/issues/browse/MB-5259
      http://www.couchbase.com/issues/browse/MB-5291

      <0.4181.3> exited with {exited,
      {'EXIT',<0.4182.3>,
      {replicator_died,
      {'EXIT',<15697.23073.1>,
      {badmatch,
      {memcached_error,auth_error,
      <<"Auth failure">>}}}}}}

      Failing testcase:-
      swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=1,num-buckets=4,num-swap=1,swap-orchestrator=True
      1) Failover a node orchestrator
      2) Swap rebalance it.

      1. 10.1.3.152-8091-diag.txt.gz
        1.76 MB
        Karan Kumar
      2. 10.1.3.153-8091-diag.txt.gz
        2.03 MB
        Karan Kumar
      3. 10.1.3.155-8091-diag.txt.gz
        1.25 MB
        Karan Kumar
      4. 10.1.3.156-8091-diag.txt.gz
        1.87 MB
        Karan Kumar
      5. 10.1.3.157-8091-diag.txt.gz
        1.94 MB
        Karan Kumar
      6. 10.1.3.158-8091-diag.txt.gz
        1.81 MB
        Karan Kumar
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #380 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/380/)
        MB-5513: don't allow ns_memcached to think it started twice (Revision e42fa1fcd59b54800da73a2702458adb2eae069d)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/ns_memcached.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #380 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/380/ ) MB-5513 : don't allow ns_memcached to think it started twice (Revision e42fa1fcd59b54800da73a2702458adb2eae069d) Result = SUCCESS Aliaksey Kandratsenka : Files : src/ns_memcached.erl
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #371 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/371/)
        MB-5513: wait isasl.pw refresh before flagging warmup completion (Revision 9f7487ec66bc5c6dc8dde0eb4347869cd96d9a79)

        Result = SUCCESS
        Aliaksey Artamonau :
        Files :

        • src/ns_config_isasl_sync.erl
        • src/ns_memcached.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #371 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/371/ ) MB-5513 : wait isasl.pw refresh before flagging warmup completion (Revision 9f7487ec66bc5c6dc8dde0eb4347869cd96d9a79) Result = SUCCESS Aliaksey Artamonau : Files : src/ns_config_isasl_sync.erl src/ns_memcached.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Done

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Done
        Hide
        karan Karan Kumar (Inactive) added a comment -

        =========================CRASH REPORT=========================
        crasher:
        initial call: ebucketmigrator_srv:init/1
        pid: <0.6065.2>
        registered_name: []
        exception error: no match of right hand side value

        {memcached_error,auth_error,<<"Auth failure">>}

        in function ebucketmigrator_srv:connect/4
        in call from ebucketmigrator_srv:init/1
        ancestors: ['ns_vbm_sup-bucket-3','single_bucket_sup-bucket-3',
        <0.3856.2>]
        messages: []
        links: Port<0.23530>,<0.5813.2>,#Port<0.23527>
        dictionary: []
        trap_exit: false
        status: running
        heap_size: 377
        stack_size: 24
        reductions: 1406
        neighbours:

        Show
        karan Karan Kumar (Inactive) added a comment - =========================CRASH REPORT========================= crasher: initial call: ebucketmigrator_srv:init/1 pid: <0.6065.2> registered_name: [] exception error: no match of right hand side value {memcached_error,auth_error,<<"Auth failure">>} in function ebucketmigrator_srv:connect/4 in call from ebucketmigrator_srv:init/1 ancestors: ['ns_vbm_sup-bucket-3','single_bucket_sup-bucket-3', <0.3856.2>] messages: [] links: Port<0.23530>,<0.5813.2>,#Port<0.23527> dictionary: [] trap_exit: false status: running heap_size: 377 stack_size: 24 reductions: 1406 neighbours:

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews