Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37167

Rebalance multiple kv nodes failed with Some apps are deploying or resuming on some or all Eventing nodes

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      build: 6.5.0-4917 , last passed on 6.5.0-4908

      • Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
      • Deploy SBM handler with timer
      • Start data loading in source bucket
      • Add 2 more kv nodes – failed
      • Verify results in source bucket
      • Remove above added nodes from the cluster

      ./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers 

      2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}
      		 
       

      Attachments

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. CCBC-1118 Libcouchbase crash observed in eventing

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. CCBC-1130 Crash in lcbvb_destroy function

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            vikas.chaudhary Vikas Chaudhary created issue -
            jeelan.poola Jeelan Poola made changes -
            Field Original Value New Value
            Assignee Jeelan Poola [ jeelan.poola ] Suraj Naik [ suraj.naik ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Suraj Naik [ suraj.naik ] Satya Nand [ satya.nand ]
            suraj.naik Suraj Naik (Inactive) made changes -
            Link This issue depends on CCBC-1118 [ CCBC-1118 ]
            jeelan.poola Jeelan Poola made changes -
            Component/s clients [ 10042 ]
            Component/s eventing [ 14026 ]
            lynn.straus Lynn Straus made changes -
            Labels functional-test approved-for-mad-hatter functional-test
            lynn.straus Lynn Straus made changes -
            Link This issue depends on CCBC-1130 [ CCBC-1130 ]
            vikas.chaudhary Vikas Chaudhary made changes -
            Description build: 6.5.0-4917 , last passed on 6.5.0-4908
            {noformat}
            ./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}
            2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}             		build: 6.5.0-4917 , last passed on 6.5.0-4908
            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}
            vikas.chaudhary Vikas Chaudhary made changes -
            Description build: 6.5.0-4917 , last passed on 6.5.0-4908
            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}             		build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * Add 2 more kv nodes
             * Verify results in source bucket
             * Remove above added nodes from the cluster 

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}
            vikas.chaudhary Vikas Chaudhary made changes -
            Description build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * Add 2 more kv nodes
             * Verify results in source bucket
             * Remove above added nodes from the cluster 

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}             		build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * Add 2 more kv nodes
             * Verify results in source bucket
             * {color:#de350b}Remove above added nodes from the cluster – failed{color}

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}
            vikas.chaudhary Vikas Chaudhary made changes -
            Description build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * Add 2 more kv nodes
             * Verify results in source bucket
             * {color:#de350b}Remove above added nodes from the cluster – failed{color}

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}             		build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * {color:#de350b}Add 2 more kv nodes – failed{color}
             * Verify results in source bucket
             * Remove above added nodes from the cluster – failed

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}
            vikas.chaudhary Vikas Chaudhary made changes -
            Description build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * {color:#de350b}Add 2 more kv nodes – failed{color}
             * Verify results in source bucket
             * Remove above added nodes from the cluster – failed

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}             		build: 6.5.0-4917 , last passed on 6.5.0-4908
             * Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
             * Deploy SBM handler with timer
             * Start data loading in source bucket
             * {color:#de350b}Add 2 more kv nodes – failed{color}
             * Verify results in source bucket
             * Remove above added nodes from the cluster

            {noformat}./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers {noformat}
            {noformat}2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}

             {noformat}
            lynn.straus Lynn Straus made changes -
            Link This issue blocks MB-36676 [ MB-36676 ]
            lynn.straus Lynn Straus made changes -
            Due Date 12/Dec/19
            ingenthr Matt Ingenthron made changes -
            Due Date 12/Dec/19 16/Dec/19
            dfinlay Dave Finlay made changes -
            Due Date 16/Dec/19 17/Dec/19
            jeelan.poola Jeelan Poola made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            vikas.chaudhary Vikas Chaudhary made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            arunkumar Arunkumar Senthilnathan made changes -
            Link This issue relates to SDKQE-1914 [ SDKQE-1914 ]

            People

              satya.nand Satya Nand (Inactive)
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty