Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6490

Rebalance failed with reason "Partition 687 not in active nor passive set" in add in node rebalance

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 2.0
    • Component/s: ns_server, view-engine
    • Security Level: Public
    • Labels:
      None
    • Environment:
      4 cores VMs CentOS, centos 6.2 64bit
      build #1653, build 2.0.0-1781

      Description

      Rebalance failed with error

      Rebalance exited with reason {{{{badmatch,
      {error,

      {error, <<"Partition 36 not in active nor passive set">>}

      }},
      [

      {capi_set_view_manager,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {gen_server,init_it,6}

      ,

      {proc_lib,init_p_do_apply,3}

      ]},
      {gen_server,call,
      ['capi_set_view_manager-saslbucket',

      {wait_index_updated,36}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-saslbucket','ns_1@10.6.2.44'}

      ,
      {if_rebalance,<0.32719.854>,
      {wait_index_updated,36}},
      infinity]}}

      with or without consistent view enable.

        • In orange cluser with build 2.0.0-1781, consistent view is enable by default and rebalance failed when add 2 nodes to cluster.
        • In Iryna cluster, consistent view is disable. She got rebalance failed with the same error as she mentioned in the following:

      index_aware_rebalance_disabled set false, 5 ddocs, 500K items
      4 nodes cluster, remove 2 nodes and add 1 node, start rebalance

      Rebalance exited with reason {{error,
      <<"Partition 687 not in active nor passive set">>},
      {gen_server,call,
      [

      {'janitor_agent-bucket-0', 'ns_1@10.3.121.120'}

      ,
      {if_rebalance,<0.14888.6>,
      {wait_index_updated,953}},
      infinity]}}

      1. 10.3.121.104-8091-diag.txt.gz
        16.80 MB
        Iryna
      2. 10.3.121.105-8091-diag.txt.gz
        16.61 MB
        Iryna
      3. 10.3.121.110-8091-diag.txt.gz
        16.22 MB
        Iryna
      4. 10.3.121.111-8091-diag.txt.gz
        15.93 MB
        Iryna
      5. 10.3.121.120-8091-diag.txt.gz
        15.72 MB
        Iryna
      6. 10.3.3.58-8091-diag.txt.gz
        15.31 MB
        Iryna
      7. 10.3.3.64-8091-diag.txt.gz
        13.99 MB
        Iryna
      8. 10.3.3.68-8091-diag.txt.gz
        15.40 MB
        Iryna
      9. 10.3.3.71-8091-diag.txt.gz
        15.34 MB
        Iryna
      10. 10.3.3.73-8091-diag.txt.gz
        15.77 MB
        Iryna
      11. 4c82d8b6-9739-40f2-885f-e2335ddb0b54-10.3.3.58-diag.txt.gz
        15.43 MB
        Iryna
      12. 4c82d8b6-9739-40f2-885f-e2335ddb0b54-10.3.3.64-diag.txt.gz
        16.23 MB
        Iryna
      13. 4c82d8b6-9739-40f2-885f-e2335ddb0b54-10.3.3.68-diag.txt.gz
        15.12 MB
        Iryna
      14. 4c82d8b6-9739-40f2-885f-e2335ddb0b54-10.3.3.71-diag.txt.gz
        14.49 MB
        Iryna
      15. 4c82d8b6-9739-40f2-885f-e2335ddb0b54-10.3.3.73-diag.txt.gz
        16.86 MB
        Iryna
      16. narrowed.txt
        522 kB
        Aleksey Kondratenko
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        iryna iryna created issue -
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        iryna iryna made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Assignee Aleksey Kondratenko [ alkondratenko ] Iryna Mironava [ iryna ]
        iryna iryna made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Assignee Iryna Mironava [ iryna ] Aleksey Kondratenko [ alkondratenko ]
        iryna iryna made changes -
        Attachment 10.3.3.58-8091-diag.txt.gz [ 14971 ]
        Attachment 10.3.3.64-8091-diag.txt.gz [ 14972 ]
        Attachment 10.3.3.68-8091-diag.txt.gz [ 14973 ]
        Attachment 10.3.3.71-8091-diag.txt.gz [ 14974 ]
        Attachment 10.3.3.73-8091-diag.txt.gz [ 14975 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Attachment narrowed.txt [ 15001 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Farshid Ghods [ farshid ]
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Farshid Ghods [ farshid ] Filipe Manana [ filipemanana ]
        farshid Farshid Ghods (Inactive) made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        thuan Thuan Nguyen made changes -
        Summary Rebalance 2 in 1 out with index_aware_rebalance_disabled=false exited with reason Partition 687 not in active nor passive set Rebalance failed with reason "Partition 687 not in active nor passive set" in add in node rebalance
        Environment 4 cores VMs CentOS
        build #1653
        4 cores VMs CentOS, centos 6.2 64bit
        build #1653, build 2.0.0-1781
        thuan Thuan Nguyen made changes -
        Description  index_aware_rebalance_disabled set false, 5 ddocs, 500K items
        4 nodes cluster, remove 2 nodes and add 1 node, start rebalance

        Rebalance exited with reason {{error,
                                          <<"Partition 687 not in active nor passive set">>},
                                      {gen_server,call,
                                          [{'janitor_agent-bucket-0',
                                               'ns_1@10.3.121.120'},
                                           {if_rebalance,<0.14888.6>,
                                               {wait_index_updated,953}},
                                           infinity]}}
        Rebalance failed with error

        Rebalance exited with reason {{{{badmatch,
        {error,
        {error,
        <<"Partition 36 not in active nor passive set">>}}},
        [{capi_set_view_manager,handle_call,3},
        {gen_server,handle_msg,5},
        {gen_server,init_it,6},
        {proc_lib,init_p_do_apply,3}]},
        {gen_server,call,
        ['capi_set_view_manager-saslbucket',
        {wait_index_updated,36},
        infinity]}},
        {gen_server,call,
        [{'janitor_agent-saslbucket','ns_1@10.6.2.44'},
        {if_rebalance,<0.32719.854>,
        {wait_index_updated,36}},
        infinity]}}

        with or without consistent view enable.

        ** In orange cluser with build 2.0.0-1781, consistent view is enable by default and rebalance failed when add 2 nodes to cluster.

        ** In Iryna cluster, consistent view is disable. She got rebalance failed with the same error as she mentioned in the following:

         index_aware_rebalance_disabled set false, 5 ddocs, 500K items
        4 nodes cluster, remove 2 nodes and add 1 node, start rebalance

        Rebalance exited with reason {{error,
                                          <<"Partition 687 not in active nor passive set">>},
                                      {gen_server,call,
                                          [{'janitor_agent-bucket-0',
                                               'ns_1@10.3.121.120'},
                                           {if_rebalance,<0.14888.6>,
                                               {wait_index_updated,953}},
                                           infinity]}}
        peter peter made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Component/s view-engine [ 10060 ]
        iryna iryna made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        FilipeManana Filipe Manana (Inactive) made changes -
        Assignee Filipe Manana [ filipemanana ] Iryna Mironava [ iryna ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Iryna Mironava [ iryna ] Aleksey Kondratenko [ alkondratenko ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        iryna iryna made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes