Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36767

[System test]: service_rebalance_failed,eventing, worker died, timed out

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build: 6.5.0-4744

      Test: longevity

      Error in 172.23.108.103 :

      2019-11-05T00:45:10.071-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.103) - Rebalance exited with reason {service_rebalance_failed,eventing,
                                    {worker_died,
                                     {'EXIT',<0.17028.638>,
                                      {rebalance_failed,
                                       {service_error,
                                        <<"eventing rebalance hasn't made progress for past 600 secs">>}}}}}.
      Rebalance Operation Id = c90499945c2ef85a2a52fd0c33723b94
      -------------------------------
       
       
      per_node_processes('ns_1@172.23.108.103') =
           {<0.32723.655>,
            [{backtrace,
                 [<<"Program counter: 0x00007f2d36703070 (misc:wait_for_process/2 + 128)">>,
                  <<"CP: 0x0000000000000000 (invalid)">>,<<"arity = 0">>,<<>>,
                  <<"0x00007f2cd32a6b40 Return addr 0x00007f2d9a54a550 (proc_lib:init_p/3 + 288)">>,
                  <<"y(0)     #Ref<0.1111397241.633602053.235983>">>,
                  <<"y(1)     infinity">>,<<>>,
                  <<"0x00007f2cd32a6b58 Return addr 0x0000000000942608 (<terminate process normally>)">>,
                  <<"y(0)     Catch 0x00007f2d9a54a570 (proc_lib:init_p/3 + 320)">>,
                  <<"y(1)     []">>,<<>>]},
             {messages,[]},
             {dictionary,
                 [{'$initial_call',
                      {compaction_daemon,'-spawn_scheduled_kv_compactor/2-fun-0-',
                          0}},
                  {'$ancestors',
                      [compaction_daemon,ns_server_sup,ns_server_nodes_sup,
                       <0.205.0>,ns_server_cluster_sup,root_sup,<0.117.0>]}]},
             {registered_name,[]},
             {status,waiting},
             {initial_call,{proc_lib,init_p,3}},
             {error_handler,error_handler},
             {garbage_collection,
                 [{max_heap_size,#{error_logger => true,kill => true,size => 0}},
                  {min_bin_vheap_size,46422},
                  {min_heap_size,233},
      
      

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.164.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.104.88.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.96.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1572944734/collectinfo-2019-11-05T090536-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankit.prabhu Ankit Prabhu
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty