Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31462

Rebalancing out all eventing node make rebalance never completes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 6.0.0
    • 6.0.0
    • eventing

    Description

      Build: 6.0.0-1673 

      When we rebalance out all the eventing nodes an add a new node then rebalance hung and it wont fail after 600sec because progress is going back and forth. Not seen with 6.0.0-1663

      Job: http://qa.sc.couchbase.com/job/test_suite_executor/94854/ 

      ./testrunner -i /tmp/testexec.27105.ini -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_rebalance_out_all_eventing_nodes_and_rebalance_in_eventing_node_and_functions_should_be_restored,nodes_init=6,services_init=kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=20,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers 
       
      
      

      [2018-10-01 03:06:53,003] - [rest_client:1615] INFO - rebalance percentage : 90.00 %
      [2018-10-01 03:06:53,003] - [task:513] INFO - Rebalance - status: running, progress: 90.00%
      [2018-10-01 03:07:03,046] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:07:03,047] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:07:13,089] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:07:13,090] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:07:23,139] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:07:23,140] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:07:33,184] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:07:33,184] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:07:43,229] - [rest_client:1615] INFO - rebalance percentage : 53.33 %
      [2018-10-01 03:07:43,230] - [task:513] INFO - Rebalance - status: running, progress: 53.33%
      [2018-10-01 03:07:53,273] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:07:53,273] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:08:03,315] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:08:03,315] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:08:13,359] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:08:13,360] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:08:23,403] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:08:23,403] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:08:33,447] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:08:33,447] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:08:43,494] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:08:43,495] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:08:53,538] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:08:53,538] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:09:03,580] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:09:03,580] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:09:13,620] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:09:13,621] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:09:23,662] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:09:23,662] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:09:33,704] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:09:33,704] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:09:43,746] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:09:43,747] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:09:53,789] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:09:53,789] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:10:03,835] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:10:03,836] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:10:13,882] - [rest_client:1615] INFO - rebalance percentage : 86.67 %
      [2018-10-01 03:10:13,882] - [task:513] INFO - Rebalance - status: running, progress: 86.67%
      [2018-10-01 03:10:23,925] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:10:23,925] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:10:33,971] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:10:33,972] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:10:44,014] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:10:44,015] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:10:54,055] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:10:54,056] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:11:04,103] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:11:04,103] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:11:14,153] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:11:14,154] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:11:24,200] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:11:24,200] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:11:34,243] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:11:34,244] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:11:44,287] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:11:44,288] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:11:54,332] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:11:54,332] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:12:04,373] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:12:04,374] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:12:14,414] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:12:14,415] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:12:24,460] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:12:24,460] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:12:34,502] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:12:34,502] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:12:44,548] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:12:44,549] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:12:54,589] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:12:54,590] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:13:04,633] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:13:04,633] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:13:14,676] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:13:14,677] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:13:24,719] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:13:24,719] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:13:34,764] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:13:34,764] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:13:44,805] - [rest_client:1615] INFO - rebalance percentage : 53.33 %
      [2018-10-01 03:13:44,806] - [task:513] INFO - Rebalance - status: running, progress: 53.33%
      [2018-10-01 03:13:54,853] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:13:54,854] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:14:04,897] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:14:04,897] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:14:14,938] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:14:14,938] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:14:24,982] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:14:24,982] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:14:35,026] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:14:35,027] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:14:45,069] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:14:45,069] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:14:55,115] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:14:55,116] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:15:05,161] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:15:05,161] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:15:15,202] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:15:15,203] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:15:25,246] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:15:25,246] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:15:35,288] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:15:35,288] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:15:45,330] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:15:45,331] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:15:55,376] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:15:55,376] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:16:05,419] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:16:05,419] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:16:15,460] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:16:15,461] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:16:25,501] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:16:25,502] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:16:35,546] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:16:35,547] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:16:45,591] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:16:45,592] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:16:55,634] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:16:55,635] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:17:05,675] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:17:05,676] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:17:15,714] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:17:15,715] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:17:25,761] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:17:25,762] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:17:35,804] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:17:35,804] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:17:45,846] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:17:45,846] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:17:55,887] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:17:55,888] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:18:05,928] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:18:05,928] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:18:15,974] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:18:15,975] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:18:26,018] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:18:26,019] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:18:36,062] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:18:36,063] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:18:46,156] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:18:46,156] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:18:56,264] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:18:56,264] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:19:06,306] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:19:06,306] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:19:16,398] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:19:16,399] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:19:26,491] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:19:26,491] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:19:36,534] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:19:36,534] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:19:46,575] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:19:46,576] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:19:56,622] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:19:56,622] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:20:06,665] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:20:06,665] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:20:16,703] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:20:16,704] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:20:26,752] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:20:26,752] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:20:36,798] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:20:36,799] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:20:46,842] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:20:46,843] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:20:56,885] - [rest_client:1615] INFO - rebalance percentage : 90.00 %
      [2018-10-01 03:20:56,885] - [task:513] INFO - Rebalance - status: running, progress: 90.00%
      [2018-10-01 03:21:06,928] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:21:06,929] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:21:16,972] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:21:16,972] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:21:27,010] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:21:27,011] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:21:37,054] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:21:37,055] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:21:47,095] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:21:47,096] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:21:57,139] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:21:57,140] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:22:07,183] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:22:07,183] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:22:17,227] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:22:17,227] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:22:27,273] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:22:27,274] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:22:37,323] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:22:37,324] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:22:47,364] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:22:47,365] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:22:57,409] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:22:57,410] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:23:07,451] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:23:07,452] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:23:17,496] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:23:17,497] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:23:27,540] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:23:27,540] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:23:37,583] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:23:37,583] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:23:47,625] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:23:47,626] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:23:57,668] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:23:57,669] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:24:07,710] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:24:07,710] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:24:17,752] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:24:17,753] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:24:27,807] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:24:27,807] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:24:37,854] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:24:37,854] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:24:47,899] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:24:47,899] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:24:57,941] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:24:57,942] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:25:07,983] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
      [2018-10-01 03:25:07,984] - [task:513] INFO - Rebalance - status: running, progress: 50.00%
      [2018-10-01 03:25:18,026] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:25:18,027] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:25:28,069] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:25:28,069] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:25:38,113] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:25:38,113] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:25:48,156] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:25:48,156] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:25:58,197] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:25:58,197] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:26:08,242] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:26:08,242] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:26:18,284] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:26:18,285] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:26:28,335] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:26:28,335] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:26:38,388] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:26:38,389] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:26:48,442] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:26:48,442] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:26:58,487] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:26:58,488] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:27:08,540] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:27:08,541] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:27:18,593] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:27:18,593] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:27:28,637] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:27:28,637] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:27:38,689] - [rest_client:1615] INFO - rebalance percentage : 86.67 %
      [2018-10-01 03:27:38,689] - [task:513] INFO - Rebalance - status: running, progress: 86.67%
      [2018-10-01 03:27:48,739] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:27:48,739] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:27:58,779] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:27:58,780] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:28:08,822] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:28:08,823] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:28:18,865] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:28:18,865] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:28:28,917] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:28:28,918] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:28:38,963] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:28:38,964] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:28:49,004] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:28:49,004] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:28:59,048] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:28:59,048] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:29:09,090] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:29:09,091] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:29:19,138] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:29:19,138] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:29:29,181] - [rest_client:1615] INFO - rebalance percentage : 53.33 %
      [2018-10-01 03:29:29,181] - [task:513] INFO - Rebalance - status: running, progress: 53.33%
      [2018-10-01 03:29:39,222] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:29:39,223] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:29:49,267] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:29:49,268] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:29:59,312] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:29:59,313] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:30:09,356] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:30:09,357] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:30:19,399] - [rest_client:1615] INFO - rebalance percentage : 86.67 %
      [2018-10-01 03:30:19,399] - [task:513] INFO - Rebalance - status: running, progress: 86.67%
      [2018-10-01 03:30:29,456] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:30:29,456] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:30:39,501] - [rest_client:1615] INFO - rebalance percentage : 56.67 %
      [2018-10-01 03:30:39,502] - [task:513] INFO - Rebalance - status: running, progress: 56.67%
      [2018-10-01 03:30:49,551] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:30:49,552] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:30:59,593] - [rest_client:1615] INFO - rebalance percentage : 83.33 %
      [2018-10-01 03:30:59,593] - [task:513] INFO - Rebalance - status: running, progress: 83.33%
      [2018-10-01 03:31:09,634] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:31:09,634] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:31:19,679] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:31:19,679] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:31:29,728] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:31:29,728] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:31:39,769] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:31:39,769] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:31:49,814] - [rest_client:1615] INFO - rebalance percentage : 53.33 %
      [2018-10-01 03:31:49,815] - [task:513] INFO - Rebalance - status: running, progress: 53.33%
      [2018-10-01 03:31:59,858] - [rest_client:1615] INFO - rebalance percentage : 86.67 %
      [2018-10-01 03:31:59,859] - [task:513] INFO - Rebalance - status: running, progress: 86.67%
      [2018-10-01 03:32:09,902] - [rest_client:1615] INFO - rebalance percentage : 86.67 %
      [2018-10-01 03:32:09,903] - [task:513] INFO - Rebalance - status: running, progress: 86.67%
      [2018-10-01 03:32:19,944] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:32:19,944] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:32:29,989] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:32:29,989] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:32:40,041] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:32:40,042] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:32:50,081] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:32:50,081] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:33:00,120] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:33:00,121] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:33:10,160] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:33:10,161] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:33:20,201] - [rest_client:1615] INFO - rebalance percentage : 63.33 %
      [2018-10-01 03:33:20,201] - [task:513] INFO - Rebalance - status: running, progress: 63.33%
      [2018-10-01 03:33:30,241] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:33:30,241] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:33:40,282] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:33:40,282] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:33:50,323] - [rest_client:1615] INFO - rebalance percentage : 76.67 %
      [2018-10-01 03:33:50,323] - [task:513] INFO - Rebalance - status: running, progress: 76.67%
      [2018-10-01 03:34:00,367] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:34:00,368] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:34:10,406] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:34:10,406] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:34:20,445] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:34:20,446] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:34:30,492] - [rest_client:1615] INFO - rebalance percentage : 70.00 %
      [2018-10-01 03:34:30,493] - [task:513] INFO - Rebalance - status: running, progress: 70.00%
      [2018-10-01 03:34:40,541] - [rest_client:1615] INFO - rebalance percentage : 66.67 %
      [2018-10-01 03:34:40,541] - [task:513] INFO - Rebalance - status: running, progress: 66.67%
      [2018-10-01 03:34:50,589] - [rest_client:1615] INFO - rebalance percentage : 90.00 %
      [2018-10-01 03:34:50,590] - [task:513] INFO - Rebalance - status: running, progress: 90.00%
      [2018-10-01 03:35:00,630] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:35:00,630] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:35:10,672] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:35:10,672] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:35:20,716] - [rest_client:1615] INFO - rebalance percentage : 60.00 %
      [2018-10-01 03:35:20,716] - [task:513] INFO - Rebalance - status: running, progress: 60.00%
      [2018-10-01 03:35:30,759] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
      [2018-10-01 03:35:30,759] - [task:513] INFO - Rebalance - status: running, progress: 50.00%
      [2018-10-01 03:35:40,800] - [rest_client:1615] INFO - rebalance percentage : 80.00 %
      [2018-10-01 03:35:40,800] - [task:513] INFO - Rebalance - status: running, progress: 80.00%
      [2018-10-01 03:35:50,840] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:35:50,840] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:36:00,881] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:36:00,882] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:36:10,920] - [rest_client:1615] INFO - rebalance percentage : 73.33 %
      [2018-10-01 03:36:10,921] - [task:513] INFO - Rebalance - status: running, progress: 73.33%
      [2018-10-01 03:36:20,967] - [rest_client:1615] INFO - rebalance percentage : 63.33 %

      Other job fails due to same is : http://qa.sc.couchbase.com/job/test_suite_executor/94685/ 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              varun.velamuri Varun Velamuri
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty