[System test]: Eventing rebalance hung

 [2019-03-13T13:57:31-07:00, sequoiatools/couchbase-cli:d1893d] server-add -c 172.23.108.103:8091 --server-add 172.23.97.239:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index

[2019-03-13T13:58:02-07:00, sequoiatools/couchbase-cli:fc956a] rebalance -c 172.23.108.103:8091 -u Administrator -p password

{"name":"ERR_REBALANCE_ONGOING","code":36,"description":"Rebalance ongoing on some/all Eventing nodes, creating new apps or changing settings for existing apps isn't allowed","attributes":null,"runtime_info":{"code":36,"info":"Rebalance ongoing on some/all Eventing nodes, creating new apps or changing settings for existing apps isn't allowed"}} {'date': 'Wed, 13 Mar 2019 20:07:20 GMT', 'status': '406', 'content-length': '346', 'content-type': 'application/json'}

#Traceback (most recent call last):

-  File "/eventing.py", line 326, in <module>

    EventingOperations().run()

'  File "/eventing.py", line 29, in run

I    response = self.perform_eventing_lifecycle_operation(app_definition)

H  File "/eventing.py", line 89, in perform_eventing_lifecycle_operation

6    raise Exception("Failed to undeploy application")

*Exception: Failed to undeploy application 

2019-03-13T23:17:52.989-07:00 [Info] Consumer::RebalanceTaskProgress [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vbsRemainingToGiveUp len: 0 dump: [] vbsRemainingToOwn len: 135 dump: [685-692, 694-699, 701-707, 709-712, 715-719, 721, 723-747, 749-750, 755-757, 759-765, 767, 769-772, 775-779, 782-786, 788-797, 799-803, 807-816, 818-819, 821-822, 825, 828, 832-835, 837-853]

2019-03-13T23:17:52.991-07:00 [Info] Consumer::RebalanceTaskProgress [worker_bucket_op_function_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_2.sock:32505] vbsRemainingToGiveUp len: 0 dump: [] vbsRemainingToOwn len: 0 dump: []

2019-03-13T23:17:52.991-07:00 [Info] ServiceMgr::getRebalanceProgress App: bucket_op_function rebalance progress from node with rest port: 8091 progress: &{135 171}

2019-03-13T23:17:52.992-07:00 [Info] util::GetProgress endpointURL: http://172.23.98.135:8096/getRebalanceProgress VbsRemainingToShuffle: 135 VbsOwnedPerPlan: 171

2019-03-13T23:17:52.992-07:00 [Info] util::GetProgress endpointURL: http://127.0.0.1:8096/getAggRebalanceProgress VbsRemainingToShuffle: 135 VbsOwnedPerPlan: 171

2019-03-13T23:17:52.997-07:00 [Info] rebalancer::gatherProgress total vbs to shuffle: 161 remaining to shuffle: 135 progress: 16.149068322981364 counter: 12 cmp: true

2019-03-13T23:17:53.001-07:00 [Info] ServiceMgr::GetCurrentTopology rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2b, 0x60}

2019-03-13T23:17:53.001-07:00 [Info] ServiceMgr::GetTaskList rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2b, 0x60}

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb takeover routine id: 0, vbs assigned len: 45 dump: [685, 688, 691, 695, 698, 702, 705, 709, 712, 717, 721, 725, 728, 731, 734, 737, 740, 743, 746, 750, 757, 761, 764, 769, 772, 777, 782, 785, 789, 792, 795, 799, 802, 808, 811, 814, 818, 822, 832, 835, 839, 842, 845, 848, 851]

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb takeover routine id: 1, vbs assigned len: 45 dump: [686, 689, 692, 696, 699, 703, 706, 710, 715, 718, 723, 726, 729, 732, 735, 738, 741, 744, 747, 755, 759, 762, 765, 770, 775, 778, 783, 786, 790, 793, 796, 800, 803, 809, 812, 815, 819, 825, 833, 837, 840, 843, 846, 849, 852]

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb takeover routine id: 2, vbs assigned len: 45 dump: [687, 690, 694, 697, 701, 704, 707, 711, 716, 719, 724, 727, 730, 733, 736, 739, 742, 745, 749, 756, 760, 763, 767, 771, 776, 779, 784, 788, 791, 794, 797, 801, 807, 810, 813, 816, 821, 828, 834, 838, 841, 844, 847, 850, 853]

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 687 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 690 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 694 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 697 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 701 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 704 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 707 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 711 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 716 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 719 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 724 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_2:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 727 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.151-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 686 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 689 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 692 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 696 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 699 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 703 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 706 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 710 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 715 skipping vbTakeover as dcp request stream already in flight

2019-03-13T23:17:53.152-07:00 [Info] Consumer::vbsStateUpdate [worker_bucket_op_function_1:takeover_r_1:/tmp/127.0.0.1:8091_worker_bucket_op_function_1.sock:32504] vb: 718 skipping vbTakeover as dcp request stream already in flight