Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29337

System Test : Rebalance in of an indexer node failed

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2542

      The longevity system test has a step to add a new indexer node to the cluster and rebalance. Rebalance fails at this step with the following error:

      Rebalance exited with reason {service_rebalance_failed,index,
      		 
      {linked_process_died,<21433.11637.7>,
      		 
      {timeout,
      		 
      {gen_server,call,
      		 
      [<21433.8347.0>,
      		 
      {call,"ServiceAPI.GetCurrentTopology",
      		 
      #Fun<json_rpc_connection.0.125340786>},
      		 
      60000]}}}}

      Error logs on 172.23.99.20 says :

      [ns_server:error,2018-04-19T03:16:15.029-07:00,ns_1@172.23.99.20:service_agent-index<0.7761.0>:service_agent:handle_info:231]Linked process <0.11637.7> died with reason {timeout,
      		 
                                                   {gen_server,call,
      		 
                                                    [<0.8347.0>,
      		 
                                                     {call,
      		 
                                                      "ServiceAPI.GetCurrentTopology",
      		 
                                                      #Fun<json_rpc_connection.0.125340786>},
      		 
                                                     60000]}}. Terminating
      		 
      [ns_server:error,2018-04-19T03:16:15.029-07:00,ns_1@172.23.99.20:service_agent-index<0.7761.0>:service_agent:terminate:260]Terminating abnormally
      		 
      [ns_server:error,2018-04-19T03:16:15.029-07:00,ns_1@172.23.99.20:service_agent-index<0.7761.0>:service_agent:terminate:265]Terminating json rpc connection for index: <0.8347.0>
      		 
      [ns_server:error,2018-04-19T03:16:15.037-07:00,ns_1@172.23.99.20:service_agent-index<0.11426.12>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<20779.23599.50>,unset_rebalancer} that doesn't match rebalancer pid undefined

      Also the indexer logs are filled with errors like the following. Not sure if that is related :

      2018-04-19T03:34:14.162-07:00 [Info] SCAN##92302 REQUEST defnId:12905143353521314278, instId:15437697085705402619, index:default/default_result_ratings_claims_pindex, type:scan, partitions:[3 5 6 15], scans: [{Low:[null] High:nil Incl:0 ScanType:range Filters:[{CompositeFilters:[{Low:null High:nil Inclusion:0}] Low:[null] High:nil Inclusion:3 ScanType:}] Equals:<nil>}], limit:9223372036854775807, consistency:any_consistency, requestId:94f92668-8337-479f-b820-e8f189b09390, groupaggr: Groups:  &{EntryKeyId:0 KeyPos:0 Expr:<nil> ExprValue:<nil>} Aggregates:  &{AggrFunc:SUM EntryKeyId:4 KeyPos:1 Expr:<nil> ExprValue:<nil> Distinct:false}  DependsOnIndexKeys [0 1] IndexKeyNames [(`default`.`result`) (`default`.`rating`) (`default`.`claim`) (meta(`default`).`id`)] NeedDecode true NeedExplode true IsLeadingGroup true
      		 
      2018-04-19T03:34:14.163-07:00 [Info] SCAN##92302 RESPONSE status:(error = Requested Partial Aggr true Not Supported For Given Scan), requestId: 94f92668-8337-479f-b820-e8f189b09390
      		 
      2018-04-19T03:34:14.164-07:00 [Error] ScanRequest::validateGroupAggr Requested Partial Aggr true Not Supported For Given Scan

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            arunkumar Arunkumar Senthilnathan (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty