Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25646

rebalance out cbas node failed: 500 error when contacting the analytics service

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • CBAS DP3
    • CBAS DP3
    • analytics, ns_server
    • None
    • 5.0.0-640-enterprise
    • Untriaged
    • Unknown
    • CX Sprint 66

    Description

      build 5.0.0-640-enterprise

      1 kv node, 1 cbas, 1 bucket

      rebalance out cbas node failed.

      seems like cbas node crashed but I'm not sure what preceded this: I implemented my tests and something wrong happened on cleanup steps
      so, I don't have exact steps to reproduce

      Rebalance exited with reason {service_rebalance_failed,cbas,
      {linked_process_died,<15651.7059.91>,
      {no_connection,"cbas-service_api"}}}
      ns_orchestrator 000
      ns_1@172.23.106.88
      2:20:30 PM   Thu Aug 10, 2017
      Starting rebalance, KeepNodes = ['ns_1@172.23.106.88'], EjectNodes = ['ns_1@172.23.105.194'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004
      ns_1@172.23.106.88
      2:19:32 PM   Thu Aug 10, 2017
      Rebalance exited with reason {service_rebalance_failed,cbas,
      {linked_process_died,<15651.743.91>,
      {no_connection,"cbas-service_api"}}}
      ns_orchestrator 000
      ns_1@172.23.106.88
      2:16:30 PM   Thu Aug 10, 2017
      Starting rebalance, KeepNodes = ['ns_1@172.23.106.88'], EjectNodes = ['ns_1@172.23.105.194'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004
      ns_1@172.23.106.88
      2:16:22 PM   Thu Aug 10, 2017
      Rebalance exited with reason {service_rebalance_failed,cbas,
      {linked_process_died,<15651.24065.90>,
      {no_connection,"cbas-service_api"}}}
      ns_orchestrator 000
      ns_1@172.23.106.88
      2:10:30 PM   Thu Aug 10, 2017
      Starting rebalance, KeepNodes = ['ns_1@172.23.106.88'], EjectNodes = ['ns_1@172.23.105.194'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004
      ns_1@172.23.106.88
      2:10:29 PM   Thu Aug 10, 2017
      Rebalance exited with reason {service_rebalance_failed,cbas,
      {linked_process_died,<15651.2699.90>,
      {no_connection,"cbas-service_api"}}}
      ns_orchestrator 000
      ns_1@172.23.106.88
      1:56:30 PM   Thu Aug 10, 2017
      Starting rebalance, KeepNodes = ['ns_1@172.23.106.88'], EjectNodes = ['ns_1@172.23.105.194'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004
      ns_1@172.23.106.88
      1:56:10 PM   Thu Aug 10, 2017
      

      ns_server:error,2017-08-10T14:17:30.640-07:00,ns_1@172.23.105.194:service_agent-cbas<0.2343.91>:service_agent:terminate:264]Terminating abnormally
      [ns_server:error,2017-08-10T14:18:30.642-07:00,ns_1@172.23.105.194:<0.3916.91>:service_agent:wait_for_connection_loop:303]No connection with label "cbas-service_api" after 60000ms. Exiting.
      [ns_server:error,2017-08-10T14:18:30.643-07:00,ns_1@172.23.105.194:service_agent-cbas<0.3915.91>:service_agent:handle_info:235]Linked process <0.3916.91> died with reason {no_connection,"cbas-service_api"}. Terminating
      

      https://s3.amazonaws.com/bugdb/jira/MB-25646/578bdec1/172.23.105.194-20170810-1321-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-25646/578bdec1/172.23.106.88-20170810-1318-diag.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty