./testrunner -i /tmp/5-centos-nodes-jython.ini durability=MAJORITY -t bucket_param_tests.bucket_param_update.BucketParamTest.test_replica_update,nodes_init=5,num_items=10000,replicas=0,doc_ops=update;create;delete,durability=MAJORITY

Test Input params:
{'cluster_name': '5-centos-nodes-jython', 'ini': '/tmp/5-centos-nodes-jython.ini', 'replicas': '0', 'durability': 'MAJORITY', 'conf_file': '/tmp/test_run.conf', 'spec': 'test_run', 'num_nodes': 4, 'logs_folder': '/data/workspace/oel6-4node-rebalance_in_jython/logs/testrunner-19-Jul-14_21-54-30/test_1', 'nodes_init': '5', 'doc_ops': 'update;create;delete', 'num_items': '10000', 'case_number': 1}
2019-07-14 21:54:31,341 | root  | WARNING | MainThread | [data_helper:<module>:30] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially

test_replica_update (bucket_param_tests.bucket_param_update.BucketParamTest) ... 2019-07-14 21:54:34,213 | test  | INFO    | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup started for test #1 test_replica_update =========

2019-07-14 21:54:35,003 | test  | INFO    | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup started for test #1 test_replica_update =========
2019-07-14 21:54:35,746 | test  | INFO    | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup finished for test #1 test_replica_update =========

2019-07-14 21:54:59,197 | infra | INFO    | MainThread | [basetestcase:tearDownEverything:302] ========== tasks in thread pool ==========
2019-07-14 21:54:59,198 | infra | INFO    | MainThread | [basetestcase:tearDownEverything:304] ==========================================
2019-07-14 21:54:59,246 | test  | INFO    | MainThread | [basetestcase:setUp:188] Initializing cluster
2019-07-14 21:54:59,441 | test  | WARNING | MainThread | [basetestcase:_initialize_nodes:344] RAM quota was defined less than 100 MB:

2019-07-14 21:55:01,293 | test  | INFO    | MainThread | [basetestcase:setUp:209] Cluster initialized
2019-07-14 21:55:01,296 | test  | INFO    | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup finished for test #1 test_replica_update =========
2019-07-14 21:55:01,329 | test  | INFO    | MainThread | [basetestcase:sleep:373] . Sleep for 5 secs ...

2019-07-14 21:55:12,979 | test  | INFO    | pool-2-thread-5 | [table_view:display:72] Rebalance Overview
+----------------+----------+--------------+
| Nodes          | Services | Status       |
+----------------+----------+--------------+
| 172.23.105.155 | [u'kv']  | Cluster node |
| 172.23.105.159 | None     | <--- IN ---  |
| 172.23.105.205 | None     | <--- IN ---  |
| 172.23.105.206 | None     | <--- IN ---  |
+----------------+----------+--------------+

2019-07-14 21:55:13,009 | test  | INFO    | pool-2-thread-5 | [task:check:309] Rebalance - status: running, progress: 0

2019-07-14 21:55:23,055 | test  | INFO    | pool-2-thread-5 | [task:check:309] Rebalance - status: none, progress: 100
2019-07-14 21:55:23,078 | test  | INFO    | pool-2-thread-5 | [task:check:368] Rebalance completed with progress: 100% in 10.1019999981 sec
2019-07-14 21:55:23,122 | test  | INFO    | MainThread | [bucket_ready_functions:async_create_bucket:99] Creating bucket: default
2019-07-14 21:55:23,196 | infra | INFO    | pool-2-thread-6 | [BucketOperations_Rest:create_bucket:301] http://172.23.105.155:8091/pools/default/buckets with param: replicaIndex=0&maxTTL=0&flushEnabled=1&compressionMode=passive&bucketType=membase&name=default&replicaNumber=0&ramQuotaMB=2135&threadsNumber=3&evictionPolicy=valueOnly
2019-07-14 21:55:23,214 | infra | INFO    | pool-2-thread-6 | [BucketOperations_Rest:create_bucket:324] 0.02 seconds to create bucket default
2019-07-14 21:55:23,216 | test  | INFO    | MainThread | [bucket_ready_functions:sleep:84] sleep for 2 secs.  ...

2019-07-14 21:55:25,305 | test  | INFO    | MainThread | [bucket_ready_functions:_wait_warmup_completed:2138] Waiting for bucket default to complete warmup

2019-07-14 21:55:27,775 | test  | INFO    | MainThread | [bucket_ready_functions:add_rbac_user:2224] **** add built-in 'cbadminbucket' user to node 172.23.105.155 ****
2019-07-14 21:55:27,877 | test  | INFO    | MainThread | [bucket_ready_functions:add_rbac_user:2228] **** add 'admin' role to 'cbadminbucket' user ****

2019-07-14 21:55:32,697 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_0_1250_create_MAJORITY are 1250
2019-07-14 21:55:32,701 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_1250_2500_create_MAJORITY are 1250
2019-07-14 21:55:32,703 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_2500_3750_create_MAJORITY are 1250
2019-07-14 21:55:32,703 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_3750_5000_create_MAJORITY are 1250
2019-07-14 21:55:32,704 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_5000_6250_create_MAJORITY are 1250
2019-07-14 21:55:32,707 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_6250_7500_create_MAJORITY are 1250
2019-07-14 21:55:32,707 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_7500_8750_create_MAJORITY are 1250
2019-07-14 21:55:32,709 | infra | INFO    | pool-2-thread-7 | [task:call:1290] Items loaded in task LoadDocumentsTask-_8750_10000_create_MAJORITY are 1250
2019-07-14 21:55:33,181 | infra | WARNING | pool-2-thread-7 | [task_manager:print_tasks_in_pool:70] Task 'DocumentsLoadGenTask_1563166530.99' not completed
2019-07-14 21:55:33,184 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_0_1250_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,184 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_1250_2500_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,184 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_2500_3750_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,186 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_3750_5000_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,187 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_5000_6250_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,187 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_6250_7500_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,187 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_7500_8750_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,188 | infra | INFO    | pool-2-thread-7 | [task:call:1323] Task 'LoadDocumentsTask-_8750_10000_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:33,431 | test  | INFO    | MainThread | [table_view:display:72] Cluster statistics
+----------------+----------+-----------------+------------+------------+----------------+---------------+--------------+---------------+-----------------------+
| Node           | Services | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_total | Swap_mem_used | Active Items | Replica Items | Version               |
+----------------+----------+-----------------+------------+------------+----------------+---------------+--------------+---------------+-----------------------+
| 172.23.105.205 | [u'kv']  | 1.76322418136   | 4201840640 | 3658358784 | 3758092288     | 0             | 0            | 0             | 6.5.0-3748-enterprise |
| 172.23.105.155 | [u'kv']  | 7.0351758794    | 4201844736 | 3632603136 | 3758092288     | 0             | 0            | 0             | 6.5.0-3748-enterprise |
| 172.23.105.206 | [u'kv']  | 52.2842639594   | 4201844736 | 3657019392 | 3758092288     | 0             | 2373         | 0             | 6.5.0-3748-enterprise |
| 172.23.105.159 | [u'kv']  | 3.27455919395   | 4201844736 | 3646951424 | 3758092288     | 0             | 0            | 0             | 6.5.0-3748-enterprise |
+----------------+----------+-----------------+------------+------------+----------------+---------------+--------------+---------------+-----------------------+

2019-07-14 21:55:33,487 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
+---------+---------+----------+-----+-------+------------+----------+-----------+
| Bucket  | Type    | Replicas | TTL | Items | RAM Quota  | RAM Used | Disk Used |
+---------+---------+----------+-----+-------+------------+----------+-----------+
| default | membase | 0        | 0   | 2373  | 8954839040 | 85925952 | 4296761   |
+---------+---------+----------+-----+-------+------------+----------+-----------+

2019-07-14 21:55:39,227 | test  | INFO    | MainThread | [bucket_ready_functions:sleep:84] sleep for 5 secs.  ...

2019-07-14 21:55:44,709 | test  | INFO    | MainThread | [bucket_param_update:setUp:39] ==========Finished Bucket_param_test setup========
2019-07-14 21:55:44,780 | test  | INFO    | MainThread | [bucket_param_update:generic_replica_update:74] Updating replica count of bucket to 1
2019-07-14 21:55:44,782 | infra | INFO    | MainThread | [BucketOperations_Rest:change_bucket_props:361] http://172.23.105.155:8091/pools/default/buckets/default with param: replicaNumber=1
2019-07-14 21:55:44,799 | infra | INFO    | MainThread | [BucketOperations_Rest:change_bucket_props:370] bucket default updated
2019-07-14 21:55:44,852 | test  | INFO    | MainThread | [table_view:display:72] Bucket statistics
+---------+---------+----------+-----+-------+------------+-----------+-----------+
| Bucket  | Type    | Replicas | TTL | Items | RAM Quota  | RAM Used  | Disk Used |
+---------+---------+----------+-----+-------+------------+-----------+-----------+
| default | membase | 1        | 0   | 10000 | 8954839040 | 116759680 | 43983929  |
+---------+---------+----------+-----+-------+------------+-----------+-----------+

2019-07-14 21:55:47,749 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_10000_11250_create_MAJORITY are 1250
2019-07-14 21:55:47,818 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_11250_12500_create_MAJORITY are 1250
2019-07-14 21:55:47,875 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_12500_13750_create_MAJORITY are 1250
2019-07-14 21:55:47,877 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_13750_15000_create_MAJORITY are 1250
2019-07-14 21:55:47,878 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_15000_16250_create_MAJORITY are 1250
2019-07-14 21:55:47,878 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_16250_17500_create_MAJORITY are 1250
2019-07-14 21:55:47,881 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_17500_18750_create_MAJORITY are 1250
2019-07-14 21:55:47,881 | infra | INFO    | pool-2-thread-9 | [task:call:1290] Items loaded in task LoadDocumentsTask-_18750_20000_create_MAJORITY are 1250
2019-07-14 21:55:47,888 | infra | WARNING | pool-2-thread-9 | [task_manager:print_tasks_in_pool:70] Task 'DocumentsLoadGenTask_1563166546.74' not completed
2019-07-14 21:55:47,888 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_10000_11250_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,888 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_11250_12500_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,890 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_12500_13750_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,891 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_13750_15000_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,891 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_15000_16250_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,891 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_16250_17500_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,892 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_17500_18750_create_MAJORITY' complete. Loaded 1250 items
2019-07-14 21:55:47,894 | infra | INFO    | pool-2-thread-9 | [task:call:1323] Task 'LoadDocumentsTask-_18750_20000_create_MAJORITY' complete. Loaded 1250 items

2019-07-14 21:55:49,305 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_5000_5625_update_MAJORITY are 625
2019-07-14 21:55:49,332 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_5625_6250_update_MAJORITY are 625
2019-07-14 21:55:49,346 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_6250_6875_update_MAJORITY are 625
2019-07-14 21:55:49,348 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_6875_7500_update_MAJORITY are 625
2019-07-14 21:55:49,349 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_7500_8125_update_MAJORITY are 625
2019-07-14 21:55:49,351 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_8125_8750_update_MAJORITY are 625
2019-07-14 21:55:49,351 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_8750_9375_update_MAJORITY are 625
2019-07-14 21:55:49,352 | infra | INFO    | pool-2-thread-8 | [task:call:1290] Items loaded in task LoadDocumentsTask-_9375_10000_update_MAJORITY are 625
2019-07-14 21:55:49,878 | test  | INFO    | pool-2-thread-7 | [table_view:display:72] Ops Trend for bucket 'default'
+----------+------------------------------------------------------------------------------------------------------+----------+
| Min      | Trend                                                                                                | Max      |
+----------+------------------------------------------------------------------------------------------------------+----------+
| 7050.000 | ...................................................................................................X | 7050.000 |
+----------+------------------------------------------------------------------------------------------------------+----------+

2019-07-14 21:55:49,880 | infra | WARNING | pool-2-thread-8 | [task_manager:print_tasks_in_pool:70] Task 'DocumentsLoadGenTask_1563166548.76' not completed
2019-07-14 21:55:49,881 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_5000_5625_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,881 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_5625_6250_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,881 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_6250_6875_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,882 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_6875_7500_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,882 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_7500_8125_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,884 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_8125_8750_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,884 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_8750_9375_update_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:49,884 | infra | INFO    | pool-2-thread-8 | [task:call:1323] Task 'LoadDocumentsTask-_9375_10000_update_MAJORITY' complete. Loaded 625 items

2019-07-14 21:55:50,701 | test  | INFO    | MainThread | [basetestcase:sleep:373] Wait for rebalance to start. Sleep for 5 secs ...
2019-07-14 21:55:51,013 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_0_625_delete_MAJORITY are 625
2019-07-14 21:55:51,025 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_625_1250_delete_MAJORITY are 625
2019-07-14 21:55:51,026 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_1250_1875_delete_MAJORITY are 625
2019-07-14 21:55:51,026 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_1875_2500_delete_MAJORITY are 625
2019-07-14 21:55:51,028 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_2500_3125_delete_MAJORITY are 625
2019-07-14 21:55:51,029 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_3125_3750_delete_MAJORITY are 625
2019-07-14 21:55:51,029 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_3750_4375_delete_MAJORITY are 625
2019-07-14 21:55:51,030 | infra | INFO    | pool-2-thread-10 | [task:call:1290] Items loaded in task LoadDocumentsTask-_4375_5000_delete_MAJORITY are 625
2019-07-14 21:55:51,048 | test  | INFO    | pool-2-thread-6 | [table_view:display:72] Rebalance Overview
+----------------+----------+--------------+
| Nodes          | Services | Status       |
+----------------+----------+--------------+
| 172.23.105.205 | [u'kv']  | Cluster node |
| 172.23.105.155 | [u'kv']  | Cluster node |
| 172.23.105.206 | [u'kv']  | Cluster node |
| 172.23.105.159 | [u'kv']  | Cluster node |
+----------------+----------+--------------+

2019-07-14 21:55:51,065 | test  | INFO    | pool-2-thread-6 | [task:check:309] Rebalance - status: running, progress: 0

2019-07-14 21:55:51,844 | test  | INFO    | pool-2-thread-4 | [table_view:display:72] Ops Trend for bucket 'default'
+----------+------------------------------------------------------------------------------------------------------+----------+
| Min      | Trend                                                                                                | Max      |
+----------+------------------------------------------------------------------------------------------------------+----------+
| 5000.000 | ...................................................................................................X | 5000.000 |
+----------+------------------------------------------------------------------------------------------------------+----------+

2019-07-14 21:55:51,845 | infra | WARNING | pool-2-thread-10 | [task_manager:print_tasks_in_pool:70] Task 'DocumentsLoadGenTask_1563166550.64' not completed
2019-07-14 21:55:51,845 | infra | WARNING | pool-2-thread-10 | [task_manager:print_tasks_in_pool:70] Task 'Rebalance_task_IN=[]_OUT=[]_1563166550.64' not completed
2019-07-14 21:55:51,846 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_0_625_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,848 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_625_1250_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,848 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_1250_1875_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,848 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_1875_2500_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,848 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_2500_3125_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,849 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_3125_3750_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,849 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_3750_4375_delete_MAJORITY' complete. Loaded 625 items
2019-07-14 21:55:51,851 | infra | INFO    | pool-2-thread-10 | [task:call:1323] Task 'LoadDocumentsTask-_4375_5000_delete_MAJORITY' complete. Loaded 625 items

2019-07-14 21:56:01,105 | test  | INFO    | pool-2-thread-6 | [task:check:309] Rebalance - status: running, progress: 0.09765625

2019-07-14 21:56:11,150 | test  | INFO    | pool-2-thread-6 | [task:check:309] Rebalance - status: running, progress: 0.09765625