Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5259

Rebalance failure due to reason replicator_died with memcached_error,auth_error,<<"Auth failure">>, also during online upgrade from 172 to 181

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.8.1-release-candidate
    • Fix Version/s: 1.8.1
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      CentOS 64 bit

      Description

      Seeing this failure, pretty much very where in regression. This is blocking running automation.

      Rebalance exited with reason {{change_filter_failed,
      {'EXIT',
      {{badmatch,failed},
      [

      {ns_vbm_sup, '-local_change_vbucket_filter/4-fun-2-',6}

      ,

      {misc,'-executing_on_new_process/1-fun-0-', 3}

      ]}}},
      [

      {ns_vbm_sup,change_vbucket_filter,4}

      ,

      {ns_vbm_sup,'-set_replicas/3-fun-2-',5}

      ,

      {lists,foldl,3}

      ,

      {ns_vbm_sup,set_replicas,3}

      ,

      {ns_vbm_sup,'-set_replicas/2-fun-1-',3}

      ,

      {lists,foreach,2}

      ,

      {ns_vbm_sup,apply_changes,2}

      ,

      {ns_vbucket_mover,sync_replicas,0}

      ]}

      1. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.92-diag.gz
        10.91 MB
        Karan Kumar
      2. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.93-diag.gz
        2.53 MB
        Karan Kumar
      3. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.94-diag.gz
        3.95 MB
        Karan Kumar
      4. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.95-diag.gz
        5.60 MB
        Karan Kumar
      5. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.97-diag.gz
        472 kB
        Karan Kumar
      6. 0ba0064f-188f-4fac-a029-0b5e0fda0d85-10.3.121.98-diag.gz
        5.78 MB
        Karan Kumar
      7. 10.1.3.114-8091-diag.txt.gz
        533 kB
        Karan Kumar
      8. 10.1.3.115-8091-diag.txt.gz
        395 kB
        Karan Kumar
      9. 10.1.3.116-8091-diag.txt.gz
        407 kB
        Karan Kumar
      10. 10.1.3.117-8091-diag.txt.gz
        228 kB
        Karan Kumar
      11. 10.1.3.118-8091-diag.txt.gz
        399 kB
        Karan Kumar
      12. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.74-diag.gz
        14.19 MB
        Karan Kumar
      13. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.75-diag.gz
        12.70 MB
        Karan Kumar
      14. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.76-diag.gz
        8.10 MB
        Karan Kumar
      15. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.77-diag.gz
        12.58 MB
        Karan Kumar
      16. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.78-diag.gz
        14.83 MB
        Karan Kumar
      17. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.79-diag.gz
        18.21 MB
        Karan Kumar
      18. 25733db2-bb7d-403e-9a65-1525b4bc6285-10.1.3.80-diag.gz
        13.95 MB
        Karan Kumar
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        karan Karan Kumar (Inactive) added a comment -

        Also hitting this when doing a online upgrade from 172 to 181.

        Failing test case:-
        upgradetests.MultipleNodeUpgradeTests.test_upgrade,initial_version=1.7.2,create_buckets=True,insert_data=True,start_upgraded_first=False,load_ratio=4,online_upgrade=True,do_new_rest=True

        The rest call for rebalance was made to the latest 181 node.

        Attaching the new set of logs.

        Show
        karan Karan Kumar (Inactive) added a comment - Also hitting this when doing a online upgrade from 172 to 181. Failing test case:- upgradetests.MultipleNodeUpgradeTests.test_upgrade,initial_version=1.7.2,create_buckets=True,insert_data=True,start_upgraded_first=False,load_ratio=4,online_upgrade=True,do_new_rest=True The rest call for rebalance was made to the latest 181 node. Attaching the new set of logs.
        Hide
        trond Trond Norbye added a comment -

        I've modified memcached to accept a special message to reread the auth file. this is pushed as http://review.couchbase.org/#change,16444 This means that ns-server needs to be updated to send the same message to memcached when it updates the isasl file (the message doesn't contain any userdata, and will spawn off a thread to update the internal sasl database. if multiple requests are sent, each of the request will spawn a new thread and the last thread finishing will win. The current implementation reads and parse the file, and at the end locks the global mutex and flip the current list of usernm/passwords).

        Reassigning to Alk to add the last bits needed to send the message from ns server (please note that you will get a reply from memcached when the operation is completed.

        Show
        trond Trond Norbye added a comment - I've modified memcached to accept a special message to reread the auth file. this is pushed as http://review.couchbase.org/#change,16444 This means that ns-server needs to be updated to send the same message to memcached when it updates the isasl file (the message doesn't contain any userdata, and will spawn off a thread to update the internal sasl database. if multiple requests are sent, each of the request will spawn a new thread and the last thread finishing will win. The current implementation reads and parse the file, and at the end locks the global mutex and flip the current list of usernm/passwords). Reassigning to Alk to add the last bits needed to send the message from ns server (please note that you will get a reply from memcached when the operation is completed .
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #358 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/358/)
        MB-5259: ask memcached to reread isasl.pw when we update the file (Revision 59444fa5f7a83d82c81d635dd0f01368fb4895d3)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/mc_client_binary.erl
        • include/mc_constants.hrl
        • src/ns_memcached.erl
        • src/ns_config_isasl_sync.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #358 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/358/ ) MB-5259 : ask memcached to reread isasl.pw when we update the file (Revision 59444fa5f7a83d82c81d635dd0f01368fb4895d3) Result = SUCCESS Aliaksey Kandratsenka : Files : src/mc_client_binary.erl include/mc_constants.hrl src/ns_memcached.erl src/ns_config_isasl_sync.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Trond's stuff is now merged

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Trond's stuff is now merged
        Hide
        karan Karan Kumar (Inactive) added a comment -

        This is fixed.

        Show
        karan Karan Kumar (Inactive) added a comment - This is fixed.

          People

          • Assignee:
            karan Karan Kumar (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: