Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47967

[System Test] Rebalance failure due to reason "badmatch, leader_activities_error, quorum_lost, lease_lost"

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1138
      Test : -test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
      Scale : 3
      Iteration : 1st

      Seeing the following in the error.log of the orchestrator node 172.23.120.58

      [ns_server:error,2021-08-14T02:16:24.487-07:00,ns_1@172.23.120.58:<0.27935.2568>:leader_activities:report_error:992]Activity {default,rebalance} failed with error {quorum_lost,
                                                      {lease_lost,
                                                       'ns_1@172.23.120.58'}}
      [ns_server:error,2021-08-14T02:16:24.989-07:00,ns_1@172.23.120.58:service_agent-eventing<0.26272.2555>:service_agent:handle_info:277]Linked process <0.30346.2555> died with reason {timeout,
                                                      {gen_server,call,
                                                       [<0.20385.2555>,
                                                        {call,
                                                         "ServiceAPI.GetCurrentTopology",
                                                         #Fun<json_rpc_connection.0.77329884>},
                                                        60000]}}. Terminating
      [ns_server:error,2021-08-14T02:16:24.989-07:00,ns_1@172.23.120.58:service_agent-eventing<0.26272.2555>:service_agent:terminate:306]Terminating abnormally
      [ns_server:error,2021-08-14T02:16:24.989-07:00,ns_1@172.23.120.58:service_agent-eventing<0.26272.2555>:service_agent:terminate:311]Terminating json rpc connection for eventing: <0.20385.2555>
      [user:error,2021-08-14T02:16:25.195-07:00,ns_1@172.23.120.58:<0.25906.2555>:ns_orchestrator:log_rebalance_completion:1416]Rebalance exited with reason {{badmatch,
                                     {leader_activities_error,
                                      {default,rebalance},
                                      {quorum_lost,
                                       {lease_lost,'ns_1@172.23.120.58'}}}},
                                    [{ns_rebalancer,rebalance,5,
                                      [{file,"src/ns_rebalancer.erl"},{line,484}]},
                                     {proc_lib,init_p_do_apply,3,
                                      [{file,"proc_lib.erl"},{line,249}]}]}.
      Rebalance Operation Id = 075e983a5d4418cc8ffd0674d013a87c
      

      172.23.120.58 is an eventing node.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chanabasappa.ghali Chanabasappa Ghali
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty