Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6493

Rebalance exits with reason "timeout, ns_memcached-bucket','ns_1@10.1.3.238'}, {get_vbucket,". Successive attempts to rebalance also fail.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0-beta
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      vBuckets: 1024
      Env: Centos 64bit

      Description

      • Create a 2:2 cluster.
      • Setup unidirectional replication of 2M items
      • Rebalance out one of the destination nodes

      Example scenario:
      Source: 10.1.3.236, 10.3.2.54
      Destination: 10.1.3.238, 10.3.2.55
      Rebalance out 10.3.2.55

      Towards the end of rebalancing on the destination, rebalance fails with the following reason:
      Rebalance exited with reason {timeout,
      {gen_server,call,
      [

      {'ns_memcached-bucket','ns_1@10.1.3.238'}

      ,

      {get_vbucket,749}

      ,
      60000]}}

      1. 10.1.3.236-8091-diag.txt.gz
        1.62 MB
        Abhinav Dangeti
      2. 10.1.3.238-8091-diag.txt.gz
        5.45 MB
        Abhinav Dangeti
      3. 10.3.2.54-8091-diag.txt.gz
        10.30 MB
        Abhinav Dangeti
      4. 10.3.2.55-8091-diag.txt.gz
        5.45 MB
        Abhinav Dangeti
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #461 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/461/)
        MB-6493: Throttle xdc_vbucket_rep initialization (Revision ff26c4fda40322ba2c0eb5d039edf5fedbc282f5)
        MB-6493: Add supervisor2 and use for restart of xdc_vbucker_rep (Revision 3f5ee80e654f7b83621e25b4f0f5c52c3a84b6b3)
        MB-6493: Fix vb map cache regression when xdcr init fails (Revision 807184854ac3c67740830dc83b5b17170282fe60)

        Result = SUCCESS
        pwansch :
        Files :

        • src/xdc_replication.erl
        • src/xdc_vbucket_rep.erl

        pwansch :
        Files :

        • src/xdc_replication.erl
        • src/xdc_vbucket_rep.erl
        • src/ns_config_default.erl
        • src/supervisor2.erl
        • include/xdc_replicator.hrl
        • src/xdc_vbucket_rep_sup.erl

        pwansch :
        Files :

        • src/xdc_vbucket_rep.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #461 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/461/ ) MB-6493 : Throttle xdc_vbucket_rep initialization (Revision ff26c4fda40322ba2c0eb5d039edf5fedbc282f5) MB-6493 : Add supervisor2 and use for restart of xdc_vbucker_rep (Revision 3f5ee80e654f7b83621e25b4f0f5c52c3a84b6b3) MB-6493 : Fix vb map cache regression when xdcr init fails (Revision 807184854ac3c67740830dc83b5b17170282fe60) Result = SUCCESS pwansch : Files : src/xdc_replication.erl src/xdc_vbucket_rep.erl pwansch : Files : src/xdc_replication.erl src/xdc_vbucket_rep.erl src/ns_config_default.erl src/supervisor2.erl include/xdc_replicator.hrl src/xdc_vbucket_rep_sup.erl pwansch : Files : src/xdc_vbucket_rep.erl
        Hide
        damien damien added a comment -

        Abhinav, can you verify if this is still a problem with my latest changes?

        Show
        damien damien added a comment - Abhinav, can you verify if this is still a problem with my latest changes?
        Hide
        abhinav Abhinav Dangeti added a comment -

        Reproduced the scenario on build 1695: Problem doesn't persist.

        Show
        abhinav Abhinav Dangeti added a comment - Reproduced the scenario on build 1695: Problem doesn't persist.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Ketaki,

        are you still running more tests to verify the fix ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - Ketaki, are you still running more tests to verify the fix ?
        Hide
        ketaki Ketaki Gangal added a comment -

        2.0-1700. Rebalance is working as expected, seeing no timeout errors.

        Show
        ketaki Ketaki Gangal added a comment - 2.0-1700. Rebalance is working as expected, seeing no timeout errors.

          People

          • Assignee:
            abhinav Abhinav Dangeti
            Reporter:
            abhinav Abhinav Dangeti
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes