2019-07-11 09:45:31,457 | root  | WARNING | MainProcess | MainThread | [data_helper:<module>:30] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially
2019-07-11 09:45:32,950 | test  | INFO    | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup started for test #1 rebalance_in_with_queries =========
2019-07-11 09:45:33,105 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:126] Deleting existing bucket default on ip:10.112.190.101 port:8091 ssh_username:root
2019-07-11 09:45:34,624 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:140] Deleted bucket: default from 10.112.190.101
2019-07-11 09:45:34,625 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:160] Waiting for bucket default deletion to finish
2019-07-11 09:45:54,950 | test  | INFO    | MainProcess | MainThread | [rest_client:monitorRebalance:1405] Rebalance done. Taken 20.0780000687 seconds to complete
2019-07-11 09:45:54,950 | test  | INFO    | MainProcess | MainThread | [rest_client:monitorRebalance:1407] Sleep for 10 seconds after rebalance
2019-07-11 09:46:05,013 | test  | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:251] CLEANUP WAS SKIPPED
2019-07-11 09:46:05,015 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:302] ========== tasks in thread pool ==========
2019-07-11 09:46:05,015 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:304] ==========================================
2019-07-11 09:46:05,038 | test  | INFO    | MainProcess | MainThread | [basetestcase:setUp:188] Initializing cluster
2019-07-11 09:46:05,263 | test  | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:344] RAM quota was defined less than 100 MB:
2019-07-11 09:46:05,874 | test  | INFO    | MainProcess | MainThread | [basetestcase:setUp:209] Cluster initialized
2019-07-11 09:46:05,875 | test  | INFO    | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:321] ========= basetestcase setup finished for test #1 rebalance_in_with_queries =========
2019-07-11 09:46:05,898 | test  | INFO    | MainProcess | MainThread | [basetestcase:sleep:373] . Sleep for 5 secs ...
2019-07-11 09:46:10,921 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:add_rbac_user:2224] **** add built-in 'cbadminbucket' user to node 10.112.190.101 ****
2019-07-11 09:46:11,065 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:add_rbac_user:2228] **** add 'admin' role to 'cbadminbucket' user ****
2019-07-11 09:46:11,122 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:async_create_bucket:99] Creating bucket: default
2019-07-11 09:46:11,157 | infra | INFO    | MainProcess | pool-2-thread-4 | [BucketOperations_Rest:create_bucket:301] http://10.112.190.101:8091/pools/default/buckets with param: replicaIndex=0&maxTTL=0&flushEnabled=1&compressionMode=off&bucketType=membase&name=default&replicaNumber=2&ramQuotaMB=529&threadsNumber=3&evictionPolicy=valueOnly
2019-07-11 09:46:11,171 | infra | INFO    | MainProcess | pool-2-thread-4 | [BucketOperations_Rest:create_bucket:324] 0.01 seconds to create bucket default
2019-07-11 09:46:11,171 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:sleep:84] sleep for 2 secs.  ...
2019-07-11 09:46:13,217 | test  | INFO    | MainProcess | MainThread | [bucket_ready_functions:_wait_warmup_completed:2138] Waiting for bucket default to complete warmup
2019-07-11 09:46:13,503 | test  | INFO    | MainProcess | MainThread | [basetestcase:sleep:373] . Sleep for 20 secs ...
2019-07-11 09:46:33,509 | infra | INFO    | MainProcess | MainThread | [cluster:async_load_gen_docs_atomicity:239] Loading documents 
2019-07-11 09:46:39,381 | test  | INFO    | MainProcess | pool-2-thread-5 | [task:call:3391] transaction timeout is 600
2019-07-11 09:46:39,401 | test  | INFO    | MainProcess | pool-2-thread-5 | [task:call:3395] transaction is com.couchbase.transactions.Transactions@3dc78382
2019-07-11 09:46:39,420 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:call:3505] Starting Atomicity load generation thread
2019-07-11 09:46:55,257 | test  | INFO    | MainProcess | pool-2-thread-6 | [task:call:3624] Atomicity Load generation thread completed
2019-07-11 09:46:55,348 | test  | INFO    | MainProcess | pool-2-thread-5 | [task:__init__:3673] the length of the keys 5000
2019-07-11 09:46:55,349 | test  | INFO    | MainProcess | pool-2-thread-5 | [task:call:3416] going to add verification task
2019-07-11 09:46:55,351 | test  | INFO    | MainProcess | pool-2-thread-7 | [task:call:3687] Starting Atomicity Verification generation thread
2019-07-11 09:46:56,836 | test  | INFO    | MainProcess | pool-2-thread-7 | [task:call:3711] Completed Atomicity Verification generation thread
2019-07-11 09:46:56,881 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Cluster statistics
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| Node           | Services | CPU_utilization | Mem_total  | Mem_free  | Swap_mem_total | Swap_mem_used | Active Items | Replica Items | Version               |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| 10.112.190.101 | [u'kv']  | 91.7525773196   | 1042685952 | 234881024 | 1107292160     | 53657600      | 5051         | 0             | 6.5.0-3687-enterprise |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+

2019-07-11 09:46:56,913 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Bucket statistics
+---------+---------+----------+-----+-------+-----------+----------+-----------+
| Bucket  | Type    | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
+---------+---------+----------+-----+-------+-----------+----------+-----------+
| default | membase | 2        | 0   | 5051  | 554696704 | 70699640 | 31053057  |
+---------+---------+----------+-----+-------+-----------+----------+-----------+

2019-07-11 09:46:56,914 | test  | INFO    | MainProcess | MainThread | [rebalance_base:setUp:66] ==========Finished rebalance base setup========
2019-07-11 09:46:57,015 | test  | ERROR   | MainProcess | pool-2-thread-8 | [rest_client:_http_request:709] GET http://10.112.190.101:8092//default/_design/dev_ddoc1 body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: not_found {"error":"not_found","reason":"missing"} auth: Administrator:password
2019-07-11 09:46:57,140 | test  | INFO    | MainProcess | pool-2-thread-8 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view0?stale=ok
2019-07-11 09:47:01,997 | test  | INFO    | MainProcess | pool-2-thread-9 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok
2019-07-11 09:47:02,013 | test  | ERROR   | MainProcess | pool-2-thread-9 | [rest_client:_http_request:709] GET http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 500 reason: unknown_error {"error":"unknown_error","reason":"view_undefined"} auth: Administrator:password
2019-07-11 09:47:02,015 | test  | INFO    | MainProcess | pool-2-thread-9 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok
2019-07-11 09:47:07,028 | test  | INFO    | MainProcess | pool-2-thread-10 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok
2019-07-11 09:47:07,046 | test  | ERROR   | MainProcess | pool-2-thread-10 | [rest_client:_http_request:709] GET http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 500 reason: unknown_error {"error":"unknown_error","reason":"view_undefined"} auth: Administrator:password
2019-07-11 09:47:07,049 | test  | INFO    | MainProcess | pool-2-thread-10 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok
2019-07-11 09:47:12,042 | test  | INFO    | MainProcess | pool-2-thread-2 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok
2019-07-11 09:47:12,061 | test  | ERROR   | MainProcess | pool-2-thread-2 | [rest_client:_http_request:709] GET http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 500 reason: unknown_error {"error":"unknown_error","reason":"view_undefined"} auth: Administrator:password
2019-07-11 09:47:12,062 | test  | INFO    | MainProcess | pool-2-thread-2 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok
2019-07-11 09:47:17,069 | test  | INFO    | MainProcess | pool-2-thread-3 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok
2019-07-11 09:47:17,088 | test  | ERROR   | MainProcess | pool-2-thread-3 | [rest_client:_http_request:709] GET http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok body:  headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 500 reason: unknown_error {"error":"unknown_error","reason":"view_undefined"} auth: Administrator:password
2019-07-11 09:47:17,089 | test  | INFO    | MainProcess | pool-2-thread-3 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok
2019-07-11 09:47:22,006 | test  | INFO    | MainProcess | pool-2-thread-1 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view0?full_set=true&connectionTimeout=60000
2019-07-11 09:47:22,028 | test  | INFO    | MainProcess | pool-2-thread-4 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?full_set=true&connectionTimeout=60000
2019-07-11 09:47:22,052 | test  | INFO    | MainProcess | pool-2-thread-6 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?full_set=true&connectionTimeout=60000
2019-07-11 09:47:22,072 | test  | INFO    | MainProcess | pool-2-thread-7 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?full_set=true&connectionTimeout=60000
2019-07-11 09:47:22,095 | test  | INFO    | MainProcess | pool-2-thread-5 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?full_set=true&connectionTimeout=60000
2019-07-11 09:47:22,154 | test  | INFO    | MainProcess | pool-2-thread-9 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view0?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:22,154 | test  | INFO    | MainProcess | pool-2-thread-1 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:22,158 | test  | INFO    | MainProcess | pool-2-thread-10 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:22,168 | test  | INFO    | MainProcess | pool-2-thread-2 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:22,171 | test  | INFO    | MainProcess | pool-2-thread-3 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:24,792 | test  | INFO    | MainProcess | pool-2-thread-2 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view2?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:24,836 | test  | INFO    | MainProcess | pool-2-thread-10 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view1?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:24,836 | test  | INFO    | MainProcess | pool-2-thread-1 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view4?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:24,838 | test  | INFO    | MainProcess | pool-2-thread-3 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view3?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:24,839 | test  | INFO    | MainProcess | pool-2-thread-9 | [rest_client:_query:468] Index query url: http://10.112.190.101:8092/default/_design/dev_ddoc1/_view/default_view0?full_set=true&stale=false&connectionTimeout=60000
2019-07-11 09:47:26,512 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Cluster statistics
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| Node           | Services | CPU_utilization | Mem_total  | Mem_free  | Swap_mem_total | Swap_mem_used | Active Items | Replica Items | Version               |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+
| 10.112.190.101 | [u'kv']  | 81.914893617    | 1042685952 | 237613056 | 1107292160     | 53657600      | 5051         | 0             | 6.5.0-3687-enterprise |
+----------------+----------+-----------------+------------+-----------+----------------+---------------+--------------+---------------+-----------------------+

2019-07-11 09:47:26,513 | test  | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:251] CLEANUP WAS SKIPPED
2019-07-11 09:47:26,513 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:302] ========== tasks in thread pool ==========
2019-07-11 09:47:26,513 | infra | INFO    | MainProcess | MainThread | [basetestcase:tearDownEverything:304] ==========================================
2019-07-11 09:47:26,513 | infra | WARNING | MainProcess | MainThread | [cluster:shutdown:605] Cluster instance shutdown with force