2019-07-24 11:26:50,878 | root  | WARNING | MainProcess | MainThread | [data_helper:<module>:30] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially
2019-07-24 11:26:53,312 | test  | INFO    | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup started for test #1 test_rebalance_in_with_ops =========
2019-07-24 11:26:53,510 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:126] Deleting existing bucket default on ip:10.112.190.101 port:8091 ssh_username:root
2019-07-24 11:26:54,986 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:140] Deleted bucket: default from 10.112.190.101
2019-07-24 11:26:54,993 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:160] Waiting for bucket default deletion to finish
2019-07-24 11:27:15,767 | test  | INFO    | MainProcess | MainThread | [rest_client:monitorRebalance:1405] Rebalance done. Taken 20.0989999771 seconds to complete
2019-07-24 11:27:15,769 | test  | INFO    | MainProcess | MainThread | [rest_client:monitorRebalance:1407] Sleep for 10 seconds after rebalance
2019-07-24 11:27:25,858 | test  | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:251] CLEANUP WAS SKIPPED
2019-07-24 11:27:25,858 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:302] ========== tasks in thread pool ==========
2019-07-24 11:27:25,859 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:304] ==========================================
2019-07-24 11:27:25,884 | test  | INFO    | MainProcess | MainThread | [basetestcase:setUp:188] Initializing cluster
2019-07-24 11:27:26,200 | test  | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:344] RAM quota was defined less than 100 MB:
2019-07-24 11:27:26,796 | test  | INFO    | MainProcess | MainThread | [basetestcase:setUp:209] Cluster initialized
2019-07-24 11:27:26,798 | test  | INFO    | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup finished for test #1 test_rebalance_in_with_ops =========
2019-07-24 11:27:26,819 | test  | INFO    | MainProcess | MainThread | [basetestcase:sleep:373] . Sleep for 5 secs ...
2019-07-24 11:27:33,706 | test  | INFO    | MainProcess | pool-2-thread-4 | [table_view:display:72] Rebalance Overview
+----------------+----------+--------------+
| Nodes          | Services | Status       |
+----------------+----------+--------------+
| 10.112.190.101 | [u'kv']  | Cluster node |
| 10.112.190.102 | None     | <--- IN ---  |
+----------------+----------+--------------+

2019-07-24 11:27:33,726 | test  | INFO    | MainProcess | pool-2-thread-4 | [task:check:309] Rebalance - status: running, progress: 0
2019-07-24 11:27:43,763 | test  | INFO    | MainProcess | pool-2-thread-4 | [task:check:309] Rebalance - status: none, progress: 100
2019-07-24 11:27:43,789 | test  | INFO    | MainProcess | pool-2-thread-4 | [task:check:368] Rebalance completed with progress: 100% in 10.0840001106 sec
2019-07-24 11:27:43,789 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:add_rbac_user:2224] **** add built-in 'cbadminbucket' user to node 10.112.190.101 ****
2019-07-24 11:27:43,924 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:add_rbac_user:2228] **** add 'admin' role to 'cbadminbucket' user ****
2019-07-24 11:27:44,019 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:async_create_bucket:99] Creating bucket: default
2019-07-24 11:27:44,098 | infra | INFO    | MainProcess | pool-2-thread-5 | [BucketOperations_Rest:create_bucket:301] http://10.112.190.101:8091/pools/default/buckets with param: replicaIndex=0&maxTTL=0&flushEnabled=1&compressionMode=off&bucketType=membase&name=default&replicaNumber=1&ramQuotaMB=100&threadsNumber=3&evictionPolicy=valueOnly
2019-07-24 11:27:44,122 | infra | INFO    | MainProcess | pool-2-thread-5 | [BucketOperations_Rest:create_bucket:324] 0.02 seconds to create bucket default
2019-07-24 11:27:44,124 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:sleep:84] sleep for 2 secs.  ...
2019-07-24 11:27:46,201 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:_wait_warmup_completed:2138] Waiting for bucket default to complete warmup
2019-07-24 11:27:47,964 | test  | INFO    | MainProcess | MainThread | [basetestcase:sleep:373] . Sleep for 20 secs ...
2019-07-24 11:28:07,970 | infra | INFO    | MainProcess | MainThread | [cluster:async_load_gen_docs_atomicity:239] Loading documents 
2019-07-24 11:28:14,042 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:call:3391] transaction timeout is 3000
2019-07-24 11:28:14,068 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:call:3395] transaction is com.couchbase.transactions.Transactions@681bd5a0
2019-07-24 11:28:14,088 | test  | INFO    | MainProcess | pool-2-thread-7 | [task:call:3504] Starting Atomicity load generation thread
2019-07-24 11:28:43,269 | test  | INFO    | MainProcess | pool-2-thread-7 | [task:call:3646] Atomicity Load generation thread completed
2019-07-24 11:29:43,384 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:__init__:3698] the length of the keys 3000
2019-07-24 11:29:43,384 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:call:3416] going to add verification task
2019-07-24 11:29:43,385 | test  | INFO    | MainProcess | pool-2-thread-8 | [task:call:3712] Starting Atomicity Verification generation thread
2019-07-24 11:29:44,642 | test  | INFO    | MainProcess | pool-2-thread-8 | [task:call:3742] Completed Atomicity Verification generation thread
2019-07-24 11:29:44,698 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Cluster statistics
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| Node           | Services | CPU_utilization | Mem_total  | Mem_free  | Swap_mem_total | Swap_mem_used | Active Items | Replica Items | Version               |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| 10.112.190.101 | [u'kv']  | 5.31914893617   | 1042685952 | 459055104 | 1107292160     | 2736128       | 1502         | 1500          | 6.5.0-3821-enterprise |
| 10.112.190.102 | [u'kv']  | 4.16666666667   | 1042685952 | 600870912 | 1107292160     | 274432        | 1500         | 1502          | 6.5.0-3821-enterprise |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+

2019-07-24 11:29:44,734 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Bucket statistics
+---------+---------+----------+-----+-------+-----------+-----------+-----------+
| Bucket  | Type    | Replicas | TTL | Items | RAM Quota | RAM Used  | Disk Used |
+---------+---------+----------+-----+-------+-----------+-----------+-----------+
| default | membase | 1        | 0   | 3002  | 209715200 | 134827600 | 81223486  |
+---------+---------+----------+-----+-------+-----------+-----------+-----------+

2019-07-24 11:29:44,734 | test  | INFO    | MainProcess | MainThread | [rebalance_base:setUp:66] ==========Finished rebalance base setup========
2019-07-24 11:29:44,757 | infra | INFO    | MainProcess | MainThread | [cluster:async_load_gen_docs_atomicity:239] Loading documents 
2019-07-24 11:29:44,980 | test  | INFO    | MainProcess | pool-2-thread-10 | [task:call:3391] transaction timeout is 3000
2019-07-24 11:29:44,982 | test  | INFO    | MainProcess | pool-2-thread-10 | [task:call:3395] transaction is com.couchbase.transactions.Transactions@3b304c06
2019-07-24 11:29:44,987 | test  | INFO    | MainProcess | pool-2-thread-2 | [task:call:3504] Starting Atomicity load generation thread
2019-07-24 11:29:47,283 | test  | INFO    | MainProcess | pool-2-thread-9 | [table_view:display:72] Rebalance Overview
+----------------+----------+--------------+
| Nodes          | Services | Status       |
+----------------+----------+--------------+
| 10.112.190.101 | [u'kv']  | Cluster node |
| 10.112.190.102 | [u'kv']  | Cluster node |
| 10.112.190.103 | None     | <--- IN ---  |
+----------------+----------+--------------+

2019-07-24 11:29:47,309 | test  | INFO    | MainProcess | pool-2-thread-9 | [task:check:309] Rebalance - status: running, progress: 0
2019-07-24 11:29:57,354 | test  | INFO    | MainProcess | pool-2-thread-9 | [task:check:309] Rebalance - status: running, progress: 4.65918237198
2019-07-24 11:30:07,434 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:_rebalance_status_and_progress:1451] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed
2019-07-24 11:30:07,497 | test  | INFO    | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2618] Latest logs from UI on 10.112.190.101:
2019-07-24 11:30:07,499 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@10.112.190.101', u'tstamp': 1563947720279L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:20.279Z', u'text': u'Rebalance exited with reason {mover_crashed,\n                              {unexpected_exit,\n                               {\'EXIT\',<0.8096.47>,\n                                {{{{{child_interrupted,\n                                     {\'EXIT\',<12951.26029.19>,socket_closed}},\n                                    [{dcp_replicator,spawn_and_wait,1,\n                                      [{file,"src/dcp_replicator.erl"},\n                                       {line,249}]},\n                                     {dcp_replicator,handle_call,3,\n                                      [{file,"src/dcp_replicator.erl"},\n                                       {line,127}]},\n                                     {gen_server,try_handle_call,4,\n                                      [{file,"gen_server.erl"},{line,636}]},\n                                     {gen_server,handle_msg,6,\n                                      [{file,"gen_server.erl"},{line,665}]},\n                                     {proc_lib,init_p_do_apply,3,\n                                      [{file,"proc_lib.erl"},{line,247}]}]},\n                                   {gen_server,call,\n                                    [<12951.26026.19>,get_partitions,\n                                     infinity]}},\n                                  {gen_server,call,\n                                   [\'dcp_replication_manager-default\',\n                                    {get_replicator_pid,846},\n                                    infinity]}},\n                                 {gen_server,call,\n                                  [{\'janitor_agent-default\',\n                                    \'ns_1@10.112.190.103\'},\n                                   {if_rebalance,<0.32070.46>,\n                                    {dcp_takeover,\'ns_1@10.112.190.102\',1001}},\n                                   infinity]}}}}}.\nRebalance Operation Id = eee33351475b2c11f9265fc562240959'}
2019-07-24 11:30:07,500 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@10.112.190.101', u'tstamp': 1563947720261L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:20.261Z', u'text': u'Worker <0.7071.47> (for action {move,{1001,\n                                      [\'ns_1@10.112.190.102\',\n                                       \'ns_1@10.112.190.101\'],\n                                      [\'ns_1@10.112.190.103\',\n                                       \'ns_1@10.112.190.101\'],\n                                      []}}) exited with reason {unexpected_exit,\n                                                                {\'EXIT\',\n                                                                 <0.8096.47>,\n                                                                 {{{{{child_interrupted,\n                                                                      {\'EXIT\',\n                                                                       <12951.26029.19>,\n                                                                       socket_closed}},\n                                                                     [{dcp_replicator,\n                                                                       spawn_and_wait,\n                                                                       1,\n                                                                       [{file,\n                                                                         "src/dcp_replicator.erl"},\n                                                                        {line,\n                                                                         249}]},\n                                                                      {dcp_replicator,\n                                                                       handle_call,\n                                                                       3,\n                                                                       [{file,\n                                                                         "src/dcp_replicator.erl"},\n                                                                        {line,\n                                                                         127}]},\n                                                                      {gen_server,\n                                                                       try_handle_call,\n                                                                       4,\n                                                                       [{file,\n                                                                         "gen_server.erl"},\n                                                                        {line,\n                                                                         636}]},\n                                                                      {gen_server,\n                                                                       handle_msg,\n                                                                       6,\n                                                                       [{file,\n                                                                         "gen_server.erl"},\n                                                                        {line,\n                                                                         665}]},\n                                                                      {proc_lib,\n                                                                       init_p_do_apply,\n                                                                       3,\n                                                                       [{file,\n                                                                         "proc_lib.erl"},\n                                                                        {line,\n                                                                         247}]}]},\n                                                                    {gen_server,\n                                                                     call,\n                                                                     [<12951.26026.19>,\n                                                                      get_partitions,\n                                                                      infinity]}},\n                                                                   {gen_server,\n                                                                    call,\n                                                                    [\'dcp_replication_manager-default\',\n                                                                     {get_replicator_pid,\n                                                                      846},\n                                                                     infinity]}},\n                                                                  {gen_server,\n                                                                   call,\n                                                                   [{\'janitor_agent-default\',\n                                                                     \'ns_1@10.112.190.103\'},\n                                                                    {if_rebalance,\n                                                                     <0.32070.46>,\n                                                                     {dcp_takeover,\n                                                                      \'ns_1@10.112.190.102\',\n                                                                      1001}},\n                                                                    infinity]}}}}'}
2019-07-24 11:30:07,503 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@10.112.190.103', u'tstamp': 1563947708064L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:08.064Z', u'text': u'Bucket "default" loaded on node \'ns_1@10.112.190.103\' in 0 seconds.'}
2019-07-24 11:30:07,505 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@10.112.190.101', u'tstamp': 1563947707276L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:07.276Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.112.190.101','ns_1@10.112.190.102',\n                                 'ns_1@10.112.190.103'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = eee33351475b2c11f9265fc562240959"}
2019-07-24 11:30:07,506 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@10.112.190.103', u'tstamp': 1563947707085L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:07.085Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'}
2019-07-24 11:30:07,506 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@10.112.190.103', u'tstamp': 1563947707036L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:55:07.036Z', u'text': u'Node ns_1@10.112.190.103 joined cluster'}
2019-07-24 11:30:07,507 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 1, u'module': u'menelaus_sup', u'type': u'info', u'node': u'ns_1@10.112.190.103', u'tstamp': 1563947706915L, u'shortText': u'web start ok', u'serverTime': u'2019-07-23T22:55:06.915Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@10.112.190.103\'. Version: "6.5.0-3821-enterprise".'}
2019-07-24 11:30:07,507 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@10.112.190.102', u'tstamp': 1563947705249L, u'shortText': u'node up', u'serverTime': u'2019-07-23T22:55:05.249Z', u'text': u"Node 'ns_1@10.112.190.102' saw that node 'ns_1@10.112.190.103' came up. Tags: []"}
2019-07-24 11:30:07,509 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@10.112.190.102', u'tstamp': 1563947584251L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:53:04.251Z', u'text': u'Bucket "default" loaded on node \'ns_1@10.112.190.102\' in 0 seconds.'}
2019-07-24 11:30:07,509 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@10.112.190.101', u'tstamp': 1563947575030L, u'shortText': u'message', u'serverTime': u'2019-07-23T22:52:55.030Z', u'text': u'Rebalance completed successfully.\nRebalance Operation Id = d55319119484de01d25801fe1d1d89f4'}
2019-07-24 11:30:07,510 | test  | ERROR   | MainProcess | pool-2-thread-9 | [task:call:219] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed
2019-07-24 11:31:01,887 | test  | INFO    | MainProcess | pool-2-thread-2 | [task:call:3646] Atomicity Load generation thread completed
2019-07-24 11:32:02,038 | test  | INFO    | MainProcess | pool-2-thread-10 | [task:__init__:3698] the length of the keys 1500
2019-07-24 11:32:02,038 | test  | INFO    | MainProcess | pool-2-thread-10 | [task:call:3416] going to add verification task
2019-07-24 11:32:02,039 | test  | INFO    | MainProcess | pool-2-thread-3 | [task:call:3712] Starting Atomicity Verification generation thread
2019-07-24 11:32:02,645 | test  | INFO    | MainProcess | pool-2-thread-3 | [task:call:3742] Completed Atomicity Verification generation thread
2019-07-24 11:32:02,655 | test  | INFO    | MainProcess | MainThread | [basetestcase:sleep:373] wait for rebalance to start. Sleep for 10 secs ...
2019-07-24 11:32:12,700 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Cluster statistics
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| Node           | Services | CPU_utilization | Mem_total  | Mem_free  | Swap_mem_total | Swap_mem_used | Active Items | Replica Items | Version               |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| 10.112.190.101 | [u'kv']  | 4.25531914894   | 1042685952 | 461668352 | 1107292160     | 3493888       | 1664         | 1814          | 6.5.0-3821-enterprise |
| 10.112.190.102 | [u'kv']  | 6.25            | 1042685952 | 566984704 | 1107292160     | 724992        | 1819         | 1758          | 6.5.0-3821-enterprise |
| 10.112.190.103 | [u'kv']  | 2.08333333333   | 1042685952 | 651923456 | 1107292160     | 8192          | 190          | 101           | 6.5.0-3821-enterprise |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+

2019-07-24 11:32:12,701 | test  | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:251] CLEANUP WAS SKIPPED
2019-07-24 11:32:12,703 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:302] ========== tasks in thread pool ==========
2019-07-24 11:32:12,703 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:304] ==========================================
2019-07-24 11:32:12,703 | infra | WARNING | MainProcess | MainThread | [cluster:shutdown:605] Cluster instance shutdown with force