Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36464

[System test]: index rebalance failed with linked_process_died

    XMLWordPrintable

Details

    Description

      Build: 6.5.0.0-4558

      Test: MH longevity with durability

      Cycle: 3rd

      Day : 2nd

      Seeing multiple rebalance failures due to linked_process_died

       [ns_server:error,2019-10-13T17:58:17.599-07:00,ns_1@172.23.104.156:service_rebalancer-index-worker<0.4369.653>:service_agent:process_bad_results:862]Service call get_node_infos (service index) failed on some nodes:
      [{'ns_1@172.23.106.100',
           {exit,
               {{linked_process_died,<25300.17420.208>,
                    {no_connection,"index-service_api"}},
                {gen_server,call,
                    [{'service_agent-index','ns_1@172.23.106.100'},
                     {if_rebalance,<0.3425.653>,get_node_info},
                     infinity]}}}}]
      [user:error,2019-10-13T17:58:17.599-07:00,ns_1@172.23.104.156:<0.23390.181>:ns_orchestrator:log_rebalance_completion:1466]Rebalance exited with reason {service_rebalance_failed,index,
                                    {agent_died,<25300.17458.208>,
                                     {linked_process_died,<25300.17420.208>,
                                      {no_connection,"index-service_api"}}}}.
      Rebalance Operation Id = 54590d3d16bdcd0d5f274dbe5ab4f68a
       
       
      [user:error,2019-10-13T18:08:27.692-07:00,ns_1@172.23.104.156:<0.23390.181>:ns_orchestrator:log_rebalance_completion:1466]Rebalance exited with reason {service_rebalance_failed,index,
                                    {agent_died,<25300.21747.209>,
                                     {linked_process_died,<25300.21722.209>,
                                      {no_connection,"index-service_api"}}}}.
      Rebalance Operation Id = f91def8ac3db712bc5a595cb4b244043

      Attachments

        For Gerrit Dashboard: MB-36464
        # Subject Branch Project Status CR V

        Activity

          The call in ns_server times out due to us not being able to connect to memcached. And that seems to be due the same issue as MB-36527, which, as far as I understand, has already been fixed.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - The call in ns_server times out due to us not being able to connect to memcached. And that seems to be due the same issue as MB-36527 , which, as far as I understand, has already been fixed.

          Feel free to assign to kv folks for confirmation, but given that the ticket was open before the fix for MB-36527 went it, I'd assume it's a duplicate.

          Aliaksey Artamonau Aliaksey Artamonau (Inactive) added a comment - - edited Feel free to assign to kv folks for confirmation, but given that the ticket was open before the fix for MB-36527 went it, I'd assume it's a duplicate.

          Build couchbase-server-6.5.0-4815 contains indexing commit 0d7a4c7 with commit message:
          MB-36464 Return if there is an error in updating Pools

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4815 contains indexing commit 0d7a4c7 with commit message: MB-36464 Return if there is an error in updating Pools

          Build couchbase-server-7.0.0-1044 contains indexing commit 0d7a4c7 with commit message:
          MB-36464 Return if there is an error in updating Pools

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1044 contains indexing commit 0d7a4c7 with commit message: MB-36464 Return if there is an error in updating Pools

          not seen on 6.5.0-4908
           

          vikas.chaudhary Vikas Chaudhary added a comment - not seen on 6.5.0-4908  

          People

            girish.benakappa Girish Benakappa
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty