<======-------> 50% EXECUTING [16s] 2021-03-23 05:01:54,161 | infra | INFO | MainThread | [task_manager:__init__:14] Initiating TaskManager with 30 threads 2021-03-23 05:01:54,168 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-03-23 05:01:54,171 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionDurabilityTests setup started for test #1 test_sync_write_in_progress ========= 2021-03-23 05:01:54,493 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:01:54,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:01:54,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:01:55,079 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:01:55,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:01:55,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:01:55,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:01:55,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:01:55,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:01:55,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:01:55,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:01:55,342 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:01:55,344 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_sync_write_in_progress ========= 2021-03-23 05:01:55,345 | test | DEBUG | MainThread | [basetestcase:setUp:292] Cleaning up cluster 2021-03-23 05:01:55,645 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3734] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 3336919, u'itemCount': 0, u'quotaPercentUsed': 1.009747362512303, u'storageTotals': {u'hdd': {u'usedByData': 47671610, u'used': 10981782281L, u'quotaTotal': 66556256256L, u'total': 66556256256L, u'free': 55241692694L}, u'ram': {u'usedByData': 59165568, u'used': 4760428544L, u'quotaTotal': 5859442688L, u'total': 8403681280L, u'quotaUsed': 5859442688L, u'quotaUsedPerNode': 2929721344L, u'quotaTotalPerNode': 2929721344L}}, u'diskUsed': 47671610, u'vbActiveNumNonResident': 0, u'memUsed': 59165568} 2021-03-23 05:01:55,648 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Deleting existing buckets [u'nJvY7ttEryDw3xOW5F_1%3kLebyP854P_iVfyU24oQnT38EeGfs6YzynIvpd0PjmRQfgde12WKbYAbYKMTzP-44-748000'] on 172.23.105.155 2021-03-23 05:01:55,648 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1575] Remove bucket nJvY7ttEryDw3xOW5F_1%3kLebyP854P_iVfyU24oQnT38EeGfs6YzynIvpd0PjmRQfgde12WKbYAbYKMTzP-44-748000 ... 2021-03-23 05:01:55,650 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1495] Deleting existing bucket nJvY7ttEryDw3xOW5F_1%3kLebyP854P_iVfyU24oQnT38EeGfs6YzynIvpd0PjmRQfgde12WKbYAbYKMTzP-44-748000 on ip:172.23.105.155 port:8091 ssh_username:root 2021-03-23 05:01:55,803 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3734] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 3336919, u'itemCount': 0, u'quotaPercentUsed': 1.009747362512303, u'storageTotals': {u'hdd': {u'usedByData': 47671610, u'used': 10981782281L, u'quotaTotal': 66556256256L, u'total': 66556256256L, u'free': 55241692694L}, u'ram': {u'usedByData': 59165568, u'used': 4760428544L, u'quotaTotal': 5859442688L, u'total': 8403681280L, u'quotaUsed': 5859442688L, u'quotaUsedPerNode': 2929721344L, u'quotaTotalPerNode': 2929721344L}}, u'diskUsed': 47671610, u'vbActiveNumNonResident': 0, u'memUsed': 59165568} 2021-03-23 05:01:56,891 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1515] Deleted bucket: nJvY7ttEryDw3xOW5F_1%3kLebyP854P_iVfyU24oQnT38EeGfs6YzynIvpd0PjmRQfgde12WKbYAbYKMTzP-44-748000 from 172.23.105.155 2021-03-23 05:01:56,892 | test | DEBUG | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:1535] Waiting for bucket nJvY7ttEryDw3xOW5F_1%3kLebyP854P_iVfyU24oQnT38EeGfs6YzynIvpd0PjmRQfgde12WKbYAbYKMTzP-44-748000 deletion to finish 2021-03-23 05:01:57,026 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Deleting existing buckets [] on 172.23.105.211 2021-03-23 05:01:57,095 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Deleting existing buckets [] on 172.23.105.212 2021-03-23 05:01:57,154 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Deleting existing buckets [] on 172.23.105.213 2021-03-23 05:01:57,263 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 120secs 2021-03-23 05:01:57,339 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:166] Rebalancing all nodes in order to remove nodes 2021-03-23 05:01:57,361 | test | DEBUG | MainThread | [rest_client:remove_nodes:124] Ejecting nodes '[u'ns_1@172.23.105.211']' from cluster 2021-03-23 05:01:57,362 | test | DEBUG | MainThread | [rest_client:rebalance:1354] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.105.155,ns_1@172.23.105.211', 'ejectedNodes': u'ns_1@172.23.105.211', 'user': 'Administrator'} 2021-03-23 05:01:57,423 | test | DEBUG | MainThread | [rest_client:rebalance:1359] Rebalance operation started 2021-03-23 05:01:57,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1594] Rebalance percentage: 0.00 % 2021-03-23 05:01:58,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1452 2021-03-23 05:01:58,022 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-03-23 05:02:01,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1598 2021-03-23 05:02:01,071 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-03-23 05:02:06,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1452 2021-03-23 05:02:06,112 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-03-23 05:02:09,137 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.7109999657 seconds to complete 2021-03-23 05:02:09,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1464 2021-03-23 05:02:09,158 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-03-23 05:02:14,236 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:213] Removed all the nodes from cluster associated with ip:172.23.105.155 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.211', 8091)] 2021-03-23 05:02:14,273 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:138] Waiting for ns_server @ 172.23.105.155:8091 2021-03-23 05:02:14,273 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-03-23 05:02:14,309 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:141] ns_server @ 172.23.105.155:8091 is running 2021-03-23 05:02:14,311 | test | WARNING | MainThread | [basetestcase:setUp:303] Cluster operation in setup will be skipped 2021-03-23 05:02:14,312 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_sync_write_in_progress ========= 2021-03-23 05:02:14,312 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_sync_write_in_progress ========= 2021-03-23 05:02:14,384 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: Rebalance_task_IN=[172.23.105.211]_OUT=[]_1616500934.31 2021-03-23 05:02:14,384 | infra | INFO | pool-1-thread-1 | [task:start_task:87] Thread 'Rebalance_task_IN=[172.23.105.211]_OUT=[]_1616500934.31' started 2021-03-23 05:02:14,384 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for Rebalance_task_IN=[172.23.105.211]_OUT=[]_1616500934.31 2021-03-23 05:02:14,388 | test | DEBUG | pool-1-thread-1 | [rest_client:add_node:1104] Adding remote node 172.23.105.211:8091 to cluster 172.23.105.155:8091 2021-03-23 05:02:17,482 | test | DEBUG | pool-1-thread-1 | [rest_client:rebalance:1354] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.105.155,ns_1@172.23.105.211', 'ejectedNodes': '', 'user': 'Administrator'} 2021-03-23 05:02:17,536 | test | DEBUG | pool-1-thread-1 | [rest_client:rebalance:1359] Rebalance operation started 2021-03-23 05:02:17,539 | test | INFO | pool-1-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.155 | kv | 7.0.0-4733-enterprise | 2.32205956588 | Cluster node | | 172.23.105.211 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-23 05:02:17,565 | test | DEBUG | pool-1-thread-1 | [rest_client:_rebalance_status_and_progress:1594] Rebalance percentage: 0.00 % 2021-03-23 05:02:17,565 | test | INFO | pool-1-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-23 05:02:17,602 | infra | DEBUG | pool-1-thread-1 | [task:check:382] Wait before next rebalance progress check 2021-03-23 05:02:17,618 | infra | DEBUG | pool-1-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L383 2021-03-23 05:02:17,618 | infra | DEBUG | pool-1-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-03-23 05:02:22,653 | infra | DEBUG | pool-1-thread-1 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1598 2021-03-23 05:02:22,654 | infra | DEBUG | pool-1-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-03-23 05:02:27,681 | test | INFO | pool-1-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-23 05:02:27,720 | test | INFO | pool-1-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 10.1809999943 sec 2021-03-23 05:02:27,720 | infra | INFO | pool-1-thread-1 | [task:complete_task:102] Thread 'Rebalance_task_IN=[172.23.105.211]_OUT=[]_1616500934.31' completed 2021-03-23 05:02:27,775 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:1983] settings/autoFailover params: failoverServerGroup=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=120&enabled=false&canAbortRebalance=false 2021-03-23 05:02:27,825 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3929] **** Add built-in 'cbadminbucket' user to node 172.23.105.155 **** 2021-03-23 05:02:28,019 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3933] **** Add 'admin' role to 'cbadminbucket' user **** 2021-03-23 05:02:28,095 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1796] Creating required buckets from template 2021-03-23 05:02:28,196 | infra | INFO | pool-1-thread-2 | [task:start_task:87] Thread 'Bucket_create_task_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' started 2021-03-23 05:02:28,196 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: Bucket_create_task_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:28,197 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for Bucket_create_task_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:28,269 | infra | INFO | pool-1-thread-2 | [BucketOperations_Rest:create_bucket:342] Creating 'couchbase' bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:28,269 | infra | DEBUG | pool-1-thread-2 | [BucketOperations_Rest:create_bucket:344] http://172.23.105.155:8091/pools/default/buckets with param: storageBackend=couchstore&viewFragmentationThreshold%255Bpercentage%255D=50&replicaIndex=1&maxTTL=0&indexCompactionMode=circular&databaseFragmentationThreshold%255Bpercentage%255D=50&flushEnabled=0¶llelDBAndViewCompaction=false&durabilityMinLevel=none&compressionMode=passive&purgeInterval=3&bucketType=couchbase&autoCompactionDefined=false&conflictResolutionType=seqno&name=JlW2lMteROclESYkprgr_%25Af8lRXCvoWklQrL-28-142000&replicaNumber=1&ramQuotaMB=2794&threadsNumber=3&evictionPolicy=valueOnly 2021-03-23 05:02:28,323 | infra | DEBUG | pool-1-thread-2 | [BucketOperations_Rest:create_bucket:368] 0.05 seconds to create bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:28,325 | test | DEBUG | Thread-1 | [task:create_scope_from_spec:3682] Creating scope for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' - _default 2021-03-23 05:02:28,328 | test | DEBUG | Thread-2 | [task:create_collection_from_spec:3711] Creating collection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:_default' - VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000 2021-03-23 05:02:28,388 | test | DEBUG | Thread-3 | [task:create_scope_from_spec:3682] Creating scope for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' - C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000 2021-03-23 05:02:28,424 | test | DEBUG | Thread-4 | [task:create_collection_from_spec:3711] Creating collection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000' - YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000 2021-03-23 05:02:28,477 | test | DEBUG | Thread-5 | [task:create_collection_from_spec:3711] Creating collection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000' - DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000 2021-03-23 05:02:28,526 | test | DEBUG | Thread-6 | [task:create_scope_from_spec:3682] Creating scope for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' - agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000 2021-03-23 05:02:28,571 | test | DEBUG | Thread-7 | [task:create_collection_from_spec:3711] Creating collection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000' - uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000 2021-03-23 05:02:28,627 | test | DEBUG | Thread-8 | [task:create_collection_from_spec:3711] Creating collection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000' - kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000 2021-03-23 05:02:28,677 | infra | INFO | pool-1-thread-2 | [task:complete_task:102] Thread 'Bucket_create_task_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' completed 2021-03-23 05:02:28,678 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2258] Waiting for all collections to be created 2021-03-23 05:02:28,799 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3734] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 0, u'itemCount': 0, u'quotaPercentUsed': 0, u'storageTotals': {u'hdd': {u'usedByData': 0, u'used': 10981782281L, u'quotaTotal': 66556256256L, u'total': 66556256256L, u'free': 55241692694L}, u'ram': {u'usedByData': 0, u'used': 4643831808L, u'quotaTotal': 5859442688L, u'total': 8403681280L, u'quotaUsed': 5859442688L, u'quotaUsedPerNode': 2929721344L, u'quotaTotalPerNode': 2929721344L}}, u'diskUsed': 0, u'vbActiveNumNonResident': 0, u'memUsed': 0} 2021-03-23 05:02:28,801 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 | couchbase | 1 | none | 0 | 0 | 5859442688 | 0 | 0 | +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-23 05:02:28,802 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-23 05:02:28,802 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:30,415 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:30,819 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:31,174 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:31,517 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:31,845 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:32,203 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:32,528 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:32,907 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:33,270 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:33,582 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4654] Performing scope/collection specific operations 2021-03-23 05:02:33,595 | test | DEBUG | MainThread | [bucket_ready_functions:perform_scope_operation:4554] Recreating scopes 2021-03-23 05:02:33,596 | test | DEBUG | MainThread | [bucket_ready_functions:perform_collection_operation:4636] Recreating collections 2021-03-23 05:02:33,598 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4744] Done Performing scope/collection specific operations 2021-03-23 05:02:33,604 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:104] Initializing mutation task for given spec 2021-03-23 05:02:33,605 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: MutateDocsFromSpecTask_1616500953.6 2021-03-23 05:02:33,607 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for MutateDocsFromSpecTask_1616500953.6 2021-03-23 05:02:33,605 | infra | INFO | pool-1-thread-3 | [task:start_task:87] Thread 'MutateDocsFromSpecTask_1616500953.6' started 2021-03-23 05:02:33,703 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500953.67 2021-03-23 05:02:33,704 | infra | INFO | pool-1-thread-4 | [task:start_task:87] Thread 'PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500953.67' started 2021-03-23 05:02:33,704 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66 2021-03-23 05:02:33,706 | infra | INFO | pool-1-thread-5 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66' started 2021-03-23 05:02:33,706 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66 2021-03-23 05:02:33,707 | infra | INFO | pool-1-thread-6 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66' started 2021-03-23 05:02:33,707 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,710 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,711 | infra | INFO | pool-1-thread-7 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67' started 2021-03-23 05:02:33,711 | infra | INFO | pool-1-thread-8 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67' started 2021-03-23 05:02:33,713 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,717 | infra | INFO | pool-1-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,713 | infra | INFO | pool-1-thread-9 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67' started 2021-03-23 05:02:33,719 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66 2021-03-23 05:02:33,717 | infra | INFO | pool-1-thread-10 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67' started 2021-03-23 05:02:33,822 | infra | INFO | pool-1-thread-8 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67' completed 2021-03-23 05:02:33,823 | infra | INFO | pool-1-thread-6 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66' completed 2021-03-23 05:02:33,823 | infra | INFO | pool-1-thread-7 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67' completed 2021-03-23 05:02:33,823 | infra | INFO | pool-1-thread-10 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67' completed 2021-03-23 05:02:33,823 | infra | INFO | pool-1-thread-9 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67' completed 2021-03-23 05:02:33,825 | infra | INFO | pool-1-thread-5 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66' completed 2021-03-23 05:02:33,825 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66 are 100 2021-03-23 05:02:33,825 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,826 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66 2021-03-23 05:02:33,826 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66 are 100 2021-03-23 05:02:33,828 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,828 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,828 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67 are 100 2021-03-23 05:02:33,829 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,829 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,829 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67 are 100 2021-03-23 05:02:33,831 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,831 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,832 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67 are 100 2021-03-23 05:02:33,832 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,832 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67 2021-03-23 05:02:33,832 | infra | DEBUG | pool-1-thread-3 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67 are 100 2021-03-23 05:02:33,834 | test | DEBUG | pool-1-thread-3 | [task:call:3799] Failed to load 0 docs from 0 to 100 2021-03-23 05:02:33,834 | infra | DEBUG | pool-1-thread-3 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500953.67 2021-03-23 05:02:34,220 | infra | DEBUG | pool-1-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:02:34,221 | infra | DEBUG | pool-1-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:02:35,223 | infra | INFO | pool-1-thread-4 | [task:complete_task:102] Thread 'PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500953.67' completed 2021-03-23 05:02:35,223 | infra | DEBUG | pool-1-thread-3 | [task:call:3815] ========= Tasks in loadgen pool======= 2021-03-23 05:02:35,224 | infra | WARNING | pool-1-thread-3 | [task_manager:print_tasks_in_pool:79] Task 'MutateDocsFromSpecTask_1616500953.6' not completed 2021-03-23 05:02:35,226 | infra | DEBUG | pool-1-thread-3 | [task:call:3817] ====================================== 2021-03-23 05:02:35,226 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create__0_100_1616500953.66' complete. Loaded 100 items 2021-03-23 05:02:35,226 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create__0_100_1616500953.66' complete. Loaded 100 items 2021-03-23 05:02:35,226 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create__0_100_1616500953.67' complete. Loaded 100 items 2021-03-23 05:02:35,227 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create__0_100_1616500953.67' complete. Loaded 100 items 2021-03-23 05:02:35,227 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000_ttl=0_create__0_100_1616500953.67' complete. Loaded 100 items 2021-03-23 05:02:35,229 | infra | DEBUG | pool-1-thread-3 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500953.6_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create__0_100_1616500953.67' complete. Loaded 100 items 2021-03-23 05:02:35,229 | infra | INFO | pool-1-thread-3 | [task:complete_task:102] Thread 'MutateDocsFromSpecTask_1616500953.6' completed 2021-03-23 05:02:35,289 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.105.155 | kv | 8.61763962598 | 4201840640 | 3360620544 | 74711040 / 3758092288 | 0 / 0 | 7.0.0-4733-enterprise | | 172.23.105.211 | kv | 16.4408079775 | 4201840640 | 3534823424 | 0 / 3758092288 | 0 / 0 | 7.0.0-4733-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-23 05:02:35,375 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:590] All nodes in cluster: [ip:172.23.105.155 port:8091 ssh_username:root, ip:172.23.105.211 port:8091 ssh_username:root] 2021-03-23 05:02:35,388 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:02:35,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:35,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:02:35,990 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:35,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:02:36,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:02:36,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:02:36,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:02:36,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:02:36,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:02:36,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:02:36,239 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:535] Starting StatsWaitTask for num_items_for_persistence on bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:36,240 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.24 2021-03-23 05:02:36,240 | infra | INFO | pool-1-thread-11 | [task:start_task:87] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.24' started 2021-03-23 05:02:36,243 | infra | DEBUG | pool-1-thread-11 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 checkpoint 2021-03-23 05:02:36,250 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:36,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:36,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:36,740 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:36,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:36,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:36,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:36,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:36,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:36,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:36,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:36,974 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:535] Starting StatsWaitTask for num_items_for_persistence on bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:36,976 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.98 2021-03-23 05:02:36,977 | infra | INFO | pool-1-thread-12 | [task:start_task:87] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.98' started 2021-03-23 05:02:36,977 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.24 2021-03-23 05:02:36,979 | infra | DEBUG | pool-1-thread-12 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 checkpoint 2021-03-23 05:02:37,818 | test | DEBUG | pool-1-thread-11 | [task:_compare:2603] Comparing 0 == 0 2021-03-23 05:02:37,819 | test | DEBUG | pool-1-thread-11 | [task:_get_checkpoint_stats_and_compare:2592] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.105.155': 0} for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:37,821 | infra | INFO | pool-1-thread-11 | [task:complete_task:102] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.24' completed 2021-03-23 05:02:37,822 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:02:37,822 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.98 2021-03-23 05:02:38,372 | test | DEBUG | pool-1-thread-12 | [task:_compare:2603] Comparing 0 == 0 2021-03-23 05:02:38,374 | test | DEBUG | pool-1-thread-12 | [task:_get_checkpoint_stats_and_compare:2592] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.105.211': 0} for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:38,375 | infra | INFO | pool-1-thread-12 | [task:complete_task:102] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_num_items_for_persistence_1616500956.98' completed 2021-03-23 05:02:38,375 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:02:38,377 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4408] Validating collection stats and item counts 2021-03-23 05:02:38,469 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:590] All nodes in cluster: [ip:172.23.105.155 port:8091 ssh_username:root, ip:172.23.105.211 port:8091 ssh_username:root] 2021-03-23 05:02:38,474 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:02:38,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:38,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:02:39,061 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:39,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:02:39,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:02:39,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:02:39,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:02:39,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:02:39,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:02:39,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:02:39,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 vbucket-seqno 2021-03-23 05:02:39,903 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:02:39,911 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:40,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:40,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:40,391 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:40,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:40,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:40,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:40,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:40,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:40,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:40,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:40,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 vbucket-seqno 2021-03-23 05:02:41,286 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:02:41,289 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2091] Verifying stats for bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:41,388 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:590] All nodes in cluster: [ip:172.23.105.155 port:8091 ssh_username:root, ip:172.23.105.211 port:8091 ssh_username:root] 2021-03-23 05:02:41,394 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:02:41,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:41,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:02:41,979 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:41,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:02:42,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:02:42,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:02:42,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:02:42,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:02:42,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:02:42,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:02:42,250 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:02:42,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:42,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:02:42,806 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:42,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:02:42,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:02:42,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:02:42,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:02:42,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:02:43,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:02:43,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:02:43,082 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:43,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:43,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:43,615 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:43,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:43,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:43,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:43,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:43,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:43,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:43,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:43,861 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:43,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:44,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:44,382 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:44,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:44,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:44,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:44,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:44,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:44,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:44,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:44,627 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:535] Starting StatsWaitTask for vb_replica_curr_items on bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:44,628 | infra | INFO | pool-1-thread-13 | [task:start_task:87] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_vb_replica_curr_items_1616500964.63' started 2021-03-23 05:02:44,628 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_vb_replica_curr_items_1616500964.63 2021-03-23 05:02:44,630 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:535] Starting StatsWaitTask for curr_items_tot on bucket JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 2021-03-23 05:02:44,630 | infra | DEBUG | pool-1-thread-13 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 all 2021-03-23 05:02:44,631 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_curr_items_tot_1616500964.63 2021-03-23 05:02:44,631 | infra | INFO | pool-1-thread-14 | [task:start_task:87] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_curr_items_tot_1616500964.63' started 2021-03-23 05:02:44,631 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_vb_replica_curr_items_1616500964.63 2021-03-23 05:02:44,632 | infra | DEBUG | pool-1-thread-14 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 all 2021-03-23 05:02:44,815 | infra | DEBUG | pool-1-thread-14 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 all 2021-03-23 05:02:44,819 | infra | DEBUG | pool-1-thread-13 | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 all 2021-03-23 05:02:44,986 | test | DEBUG | pool-1-thread-13 | [task:_compare:2603] Comparing 600 == 600 2021-03-23 05:02:44,987 | test | DEBUG | pool-1-thread-13 | [task:_get_all_stats_and_compare:2555] Ready: vb_replica_curr_items 600 == 600. Received: {'172.23.105.211': '300', '172.23.105.155': '300'} for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:44,987 | infra | INFO | pool-1-thread-13 | [task:complete_task:102] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_vb_replica_curr_items_1616500964.63' completed 2021-03-23 05:02:44,989 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_curr_items_tot_1616500964.63 2021-03-23 05:02:44,996 | test | DEBUG | pool-1-thread-14 | [task:_compare:2603] Comparing 1200 == 1200 2021-03-23 05:02:44,997 | test | DEBUG | pool-1-thread-14 | [task:_get_all_stats_and_compare:2555] Ready: curr_items_tot 1200 == 1200. Received: {'172.23.105.211': '600', '172.23.105.155': '600'} for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:44,997 | infra | INFO | pool-1-thread-14 | [task:complete_task:102] Thread 'StatsWaitTask_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_curr_items_tot_1616500964.63' completed 2021-03-23 05:02:44,999 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:02:45,000 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:02:45,000 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:02:45,000 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:02:45,141 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:590] All nodes in cluster: [ip:172.23.105.155 port:8091 ssh_username:root, ip:172.23.105.211 port:8091 ssh_username:root] 2021-03-23 05:02:45,147 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:02:45,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:45,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:02:45,693 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:45,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:02:45,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:02:45,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:02:45,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:02:45,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:02:45,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:02:45,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:02:45,961 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:46,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:46,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:46,453 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:46,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:46,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:46,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:46,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:46,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:46,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:46,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:46,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 collections 2021-03-23 05:02:46,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 collections 2021-03-23 05:02:47,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 scopes 2021-03-23 05:02:47,229 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:02:47,230 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:02:47,290 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3734] Stats: {u'opsPerSec': 30, u'diskFetches': 0, u'dataUsed': 8794856, u'itemCount': 600, u'quotaPercentUsed': 1.040651052443573, u'storageTotals': {u'hdd': {u'usedByData': 28177894, u'used': 10981782281L, u'quotaTotal': 66556256256L, u'total': 66556256256L, u'free': 55241692694L}, u'ram': {u'usedByData': 60976352, u'used': 4624678912L, u'quotaTotal': 5859442688L, u'total': 8403681280L, u'quotaUsed': 5859442688L, u'quotaUsedPerNode': 2929721344L, u'quotaTotalPerNode': 2929721344L}}, u'diskUsed': 28177894, u'vbActiveNumNonResident': 0, u'memUsed': 60976352} 2021-03-23 05:02:47,292 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 | couchbase | 1 | none | 0 | 600 | 5859442688 | 60976352 | 28177894 | +-------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-23 05:02:47,292 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_sync_write_in_progress ========= 2021-03-23 05:02:47,293 | test | INFO | MainThread | [collection_crud_negative:_CollectionDurabilityTests__get_d_level_and_error_to_simulate:54] Testing with durability_level=MAJORITY_AND_PERSIST_TO_ACTIVE, simulate_error=stop_persistence 2021-03-23 05:02:47,295 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:243] Creating SDK connection for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:02:47,608 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:02:47,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:02:47,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:02:48,114 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:02:48,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:02:48,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:02:48,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:02:48,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:02:48,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:02:48,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:02:48,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:02:48,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 vbucket-seqno 2021-03-23 05:02:48,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 vbucket 2021-03-23 05:02:49,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 vbucket 2021-03-23 05:02:49,305 | test | INFO | MainThread | [cb_error:create:39] Simulating 'stop_persistence' in 172.23.105.211 2021-03-23 05:02:49,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbepctl localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 stop 2021-03-23 05:02:49,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/epengine/collection_crud_negative.py -> test_sync_write_in_progress():L413 2021-03-23 05:02:49,957 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 3 seconds. Reason: Wait for error simulation to take effect 2021-03-23 05:02:52,959 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4654] Performing scope/collection specific operations 2021-03-23 05:02:52,969 | test | DEBUG | MainThread | [bucket_ready_functions:perform_scope_operation:4554] Recreating scopes 2021-03-23 05:02:52,969 | test | DEBUG | MainThread | [bucket_ready_functions:perform_collection_operation:4636] Recreating collections 2021-03-23 05:02:52,970 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4744] Done Performing scope/collection specific operations 2021-03-23 05:02:52,996 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:104] Initializing mutation task for given spec 2021-03-23 05:02:52,999 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: MutateDocsFromSpecTask_1616500973.0 2021-03-23 05:02:52,999 | infra | INFO | pool-1-thread-15 | [task:start_task:87] Thread 'MutateDocsFromSpecTask_1616500973.0' started 2021-03-23 05:02:53,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/epengine/collection_crud_negative.py -> test_sync_write_in_progress():L423 2021-03-23 05:02:53,029 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait for doc ops to reach server 2021-03-23 05:02:53,062 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500973.04 2021-03-23 05:02:53,062 | infra | INFO | pool-1-thread-16 | [task:start_task:87] Thread 'PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500973.04' started 2021-03-23 05:02:53,063 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 2021-03-23 05:02:53,063 | infra | INFO | pool-1-thread-17 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' started 2021-03-23 05:02:53,065 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 2021-03-23 05:02:53,065 | infra | INFO | pool-1-thread-18 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' started 2021-03-23 05:02:53,065 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:02:53,066 | infra | INFO | pool-1-thread-19 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' started 2021-03-23 05:02:53,068 | infra | INFO | pool-1-thread-20 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' started 2021-03-23 05:02:53,066 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:02:53,072 | infra | INFO | pool-1-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:02:53,072 | infra | INFO | pool-1-thread-21 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' started 2021-03-23 05:02:53,072 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 2021-03-23 05:02:53,760 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:02:53,766 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:02:53,766 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:02:55,424 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:02:55,427 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:02:55,427 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:02:57,134 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:02:57,138 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:02:57,138 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:02:58,036 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS)","requestId":1360,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"_default","documentId":"test_collections-100","opaque":"0x1cf1","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":1178,"encodingMicros":611,"totalMicros":2808,"serverMicros":0}} 2021-03-23 05:02:58,813 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:02:58,816 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:02:58,818 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:00,051 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"TIMEOUT","requestId":1362,"requestType":"InsertRequest","retried":13,"retryReasons":["KV_SYNC_WRITE_IN_PROGRESS"],"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"_default","documentId":"test_collections-100","opaque":"0x1cf3","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":844,"encodingMicros":47,"totalMicros":2002563,"serverMicros":0}} 2021-03-23 05:03:00,058 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1421,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000","documentId":"test_collections-100","opaque":"0x1d2e","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":88,"totalMicros":310}} 2021-03-23 05:03:00,058 | test | ERROR | MainThread | [basetestcase:log_failure:620] Invalid retry reason for test_collections-100: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1421,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000","documentId":"test_collections-100","opaque":"0x1d2e","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":88,"totalMicros":310}} 2021-03-23 05:03:00,437 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:03:00,441 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:03:00,443 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:02,069 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"TIMEOUT","requestId":1423,"requestType":"InsertRequest","retried":13,"retryReasons":["KV_SYNC_WRITE_IN_PROGRESS"],"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000","documentId":"test_collections-100","opaque":"0x1d30","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":1065,"encodingMicros":29,"totalMicros":2006273,"serverMicros":0}} 2021-03-23 05:03:02,078 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS)","requestId":1485,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000","documentId":"test_collections-100","opaque":"0x1d6e","scope":"C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":1358,"encodingMicros":65,"totalMicros":1703,"serverMicros":0}} 2021-03-23 05:03:02,138 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 0.000000 2021-03-23 05:03:02,141 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:03:02,142 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:03,834 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 2.700000 2021-03-23 05:03:03,836 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:03:03,838 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:04,088 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"TIMEOUT","requestId":1487,"requestType":"InsertRequest","retried":13,"retryReasons":["KV_SYNC_WRITE_IN_PROGRESS"],"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000","documentId":"test_collections-100","opaque":"0x1d70","scope":"C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":1200,"encodingMicros":24,"totalMicros":2005741,"serverMicros":0}} 2021-03-23 05:03:04,094 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS)","requestId":1549,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000","documentId":"test_collections-100","opaque":"0x1dae","scope":"agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":640,"encodingMicros":54,"totalMicros":912,"serverMicros":0}} 2021-03-23 05:03:05,522 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 2.700000 2021-03-23 05:03:05,526 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:03:05,528 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:06,108 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"TIMEOUT","requestId":1551,"requestType":"InsertRequest","retried":13,"retryReasons":["KV_SYNC_WRITE_IN_PROGRESS"],"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000","documentId":"test_collections-100","opaque":"0x1db0","scope":"agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":953,"encodingMicros":24,"totalMicros":2010435,"serverMicros":0}} 2021-03-23 05:03:06,114 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (KV_SYNC_WRITE_IN_PROGRESS)","requestId":1609,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000","documentId":"test_collections-100","opaque":"0x1dea","scope":"agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":595,"encodingMicros":59,"totalMicros":851,"serverMicros":0}} 2021-03-23 05:03:07,267 | test | DEBUG | pool-1-thread-16 | [task:record_bucket_ops:3449] Ops rate for 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000': 2.700000 2021-03-23 05:03:07,270 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3494 2021-03-23 05:03:07,270 | infra | DEBUG | pool-1-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-03-23 05:03:08,128 | test | DEBUG | MainThread | [sdk_client3:insert:772] Request cancelled/timed-out: com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"TIMEOUT","requestId":1611,"requestType":"InsertRequest","retried":13,"retryReasons":["KV_SYNC_WRITE_IN_PROGRESS"],"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000","documentId":"test_collections-100","opaque":"0x1dec","scope":"agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"dispatchMicros":1000,"encodingMicros":26,"totalMicros":2010742,"serverMicros":0}} 2021-03-23 05:03:08,134 | test | INFO | MainThread | [cb_error:revert:68] Reverting 'stop_persistence' in 172.23.105.211 2021-03-23 05:03:08,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: /opt/couchbase/bin/cbepctl localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 start 2021-03-23 05:03:08,255 | infra | INFO | pool-1-thread-20 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' completed 2021-03-23 05:03:08,255 | infra | INFO | pool-1-thread-18 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' completed 2021-03-23 05:03:08,257 | infra | INFO | pool-1-thread-17 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' completed 2021-03-23 05:03:08,257 | infra | INFO | pool-1-thread-21 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' completed 2021-03-23 05:03:08,257 | infra | INFO | pool-1-thread-19 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' completed 2021-03-23 05:03:08,257 | infra | DEBUG | pool-1-thread-15 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 are 1 2021-03-23 05:03:08,259 | test | DEBUG | pool-1-thread-15 | [task:call:3799] Failed to load 0 docs from 100 to 101 2021-03-23 05:03:08,259 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 2021-03-23 05:03:08,259 | infra | DEBUG | pool-1-thread-15 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03 are 1 2021-03-23 05:03:08,259 | test | DEBUG | pool-1-thread-15 | [task:call:3799] Failed to load 0 docs from 100 to 101 2021-03-23 05:03:08,260 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:03:08,260 | infra | DEBUG | pool-1-thread-15 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 are 1 2021-03-23 05:03:08,262 | test | DEBUG | pool-1-thread-15 | [task:call:3799] Failed to load 0 docs from 100 to 101 2021-03-23 05:03:08,262 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:03:08,262 | infra | DEBUG | pool-1-thread-15 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 are 1 2021-03-23 05:03:08,263 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for MutateDocsFromSpecTask_1616500973.0 2021-03-23 05:03:08,263 | test | DEBUG | pool-1-thread-15 | [task:call:3799] Failed to load 0 docs from 100 to 101 2021-03-23 05:03:08,263 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 2021-03-23 05:03:08,263 | infra | DEBUG | pool-1-thread-15 | [task:call:3770] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04 are 1 2021-03-23 05:03:08,263 | test | DEBUG | pool-1-thread-15 | [task:call:3799] Failed to load 0 docs from 100 to 101 2021-03-23 05:03:08,265 | infra | DEBUG | pool-1-thread-15 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500973.04 2021-03-23 05:03:08,273 | test | INFO | pool-1-thread-16 | [table_view:display:72] Ops trend for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' +-------+---------------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-------+ | 0.000 | ******....................................................................................................**X | 2.700 | +-------+---------------------------------------------------------------------------------------------------------------+-------+ 2021-03-23 05:03:08,273 | infra | INFO | pool-1-thread-16 | [task:complete_task:102] Thread 'PrintBucketStats_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_1616500973.04' completed 2021-03-23 05:03:08,275 | infra | DEBUG | pool-1-thread-15 | [task:call:3815] ========= Tasks in loadgen pool======= 2021-03-23 05:03:08,275 | infra | WARNING | pool-1-thread-15 | [task_manager:print_tasks_in_pool:79] Task 'MutateDocsFromSpecTask_1616500973.0' not completed 2021-03-23 05:03:08,275 | infra | DEBUG | pool-1-thread-15 | [task:call:3817] ====================================== 2021-03-23 05:03:08,276 | infra | DEBUG | pool-1-thread-15 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000_YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' complete. Loaded 1 items 2021-03-23 05:03:08,276 | infra | DEBUG | pool-1-thread-15 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default_VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.03' complete. Loaded 1 items 2021-03-23 05:03:08,276 | infra | DEBUG | pool-1-thread-15 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' complete. Loaded 1 items 2021-03-23 05:03:08,276 | infra | DEBUG | pool-1-thread-15 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000_agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000_uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' complete. Loaded 1 items 2021-03-23 05:03:08,278 | infra | DEBUG | pool-1-thread-15 | [task:call:3820] Task 'LoadDocs_MutateDocsFromSpecTask_1616500973.0_JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000__default__default_ttl=0_create_MAJORITY_AND_PERSIST_TO_ACTIVE_100_101_1616500973.04' complete. Loaded 1 items 2021-03-23 05:03:08,278 | infra | INFO | pool-1-thread-15 | [task:complete_task:102] Thread 'MutateDocsFromSpecTask_1616500973.0' completed 2021-03-23 05:03:08,279 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:08,338 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown FAIL 2021-03-23 05:03:08,346 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:03:08,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:03:08,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:03:08,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:03:08,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:03:08,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:03:08,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:03:09,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:03:09,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:03:09,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:03:09,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:03:09,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000 all 2021-03-23 05:03:09,355 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000, Active Resident ratio(DGM): 100% 2021-03-23 05:03:09,357 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000, Replica Resident ratio(DGM): 100% 2021-03-23 05:03:09,357 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:_default:_default 2021-03-23 05:03:09,427 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:_default:VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000 2021-03-23 05:03:09,503 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000:YH8aGmUHFqEr3Mk%RH4wYbKPz5bCULnr-RuDhbIgAvhc%IEqdQjI6R7R1UFcYWuybvvGLgOQKor6mAzRQEDCnHPsUQ4O9UVBD1TToKAiaHEIUl6h87DjXlFAwrHDkCbIrkv4gnAwt7ZXAE4m-R8eUGRxNPMMsgJYhF9DF04zKiuhgbK%tvcXBnj7LRr9flzzI6c3PvMoJ95HrBt1QV77-28-147000 2021-03-23 05:03:09,568 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000:DD-I4UcTJcfe2fSTQ1vYxPfP5VIx%-yM_oaWT_PFKzW0D%_MALrIc8yo58dAlqUwX3PJ1u6Jsw4GJxT173X7dsuor1jNYvn8hu9yqubSIvTa96PHKN6gsuEh0iaMzpSxeQsq1Wl31Y-G64_EfOyu81XsccC%ie3vKXRgd0ndysid8D7HA_EXlBcS2-28-146000 2021-03-23 05:03:09,634 | test | DEBUG | MainThread | [bucket_ready_functions:drop_scope:1121] Dropping Scope JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:C5xHG8kn%Xaj%rZlBELMTFx2K%a2Kc6PEmNiuDn6-yB97JvJKt-CrwXyyyqLDDmqpb65k8qdJbDwWYPCJO4jhOKXKFvgAyrL56K-oQnUJ7ctWGk0riK7fMlNcwDXW7m-28-145000 2021-03-23 05:03:09,697 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000:uiLdhjJj11NvE04Z1ZFUbxlyxE35bTV4K6gw1%vuCTr8Dt%1vcfji0R%75erdmWTYjHQC4kySxlYgerKpis_FPCYSJt1VCk9Nlv8qNU3vuig26Iv1TutudSkT6eAz3yjF9u14ocznaB_01ZcjlZp%JbaeN8OZnICLROBOxHVgXVGhSoIZeDn9WjqEn9T-8cIgWeDo-28-150000 2021-03-23 05:03:09,763 | test | DEBUG | MainThread | [bucket_ready_functions:drop_collection:886] Dropping Collection JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000:kj_spHvxxVYZfZSNKCjUCP4Q_KnmqR1M3Z2hg8u-28-150000 2021-03-23 05:03:09,825 | test | DEBUG | MainThread | [bucket_ready_functions:drop_scope:1121] Dropping Scope JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000:agBu0gUx10COBgm4XeHgGjyVcvODNHZi1Py6-NeHDMH9obnY6XY41YY3sbmkDMVUyXkW4p6lJaYLh0JXIXSlW_cYap8fPi%p8N6jjAjm1VwHyIh15QTq0nFfudVsX2Yar94hv8-2ElVrPE2KmH_NTvyNCMn7w9VVg11wRSPq0aTUmOLDTyvWXSuG-rdQMwssDN2Nsz-28-149000 2021-03-23 05:03:09,885 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:03:09,887 | infra | INFO | MainThread | [task_manager:shutdown:61] Running TaskManager shutdown 2021-03-23 05:03:09,888 | infra | INFO | MainThread | [task_manager:shutdown:61] Running TaskManager shutdown 2021-03-23 05:03:09,888 | infra | WARNING | MainThread | [cluster:shutdown:676] Cluster instance shutdown with force 2021-03-23 05:03:09,890 | infra | DEBUG | MainThread | [sdk_client3:shutdown:89] Closing clients from SDKClientPool 2021-03-23 05:03:09,890 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:09,928 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:09,930 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:09,969 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:09,970 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,003 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,005 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,036 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,038 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,071 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,072 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,104 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,105 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,134 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,135 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,163 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,164 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,190 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,190 | test | DEBUG | MainThread | [sdk_client3:close:330] Closing SDK for bucket 'JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000' 2021-03-23 05:03:10,217 | test | DEBUG | MainThread | [sdk_client3:close:334] Cluster disconnected and env shutdown 2021-03-23 05:03:10,223 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.155 with username: root, password: couchbase 2021-03-23 05:03:10,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:03:10,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /etc/ 2021-03-23 05:03:10,755 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:03:10,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /cygdrive/c/Windows 2021-03-23 05:03:10,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: uname -m 2021-03-23 05:03:10,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/cpuinfo 2021-03-23 05:03:10,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: df -Th 2021-03-23 05:03:10,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: cat /proc/meminfo 2021-03-23 05:03:10,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname 2021-03-23 05:03:10,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: hostname -d 2021-03-23 05:03:11,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-03-23 05:03:11,055 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:727] 172.23.105.155: No crash files found 2021-03-23 05:03:11,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-03-23 05:03:11,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:11,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:11,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:11,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:11,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:11,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:11,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-03-23 05:03:11,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-03-23 05:03:11,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:11,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:11,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-03-23 05:03:11,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-03-23 05:03:11,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:11,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:11,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-03-23 05:03:11,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-03-23 05:03:11,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:12,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:12,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-03-23 05:03:12,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-03-23 05:03:12,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:12,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:12,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-03-23 05:03:12,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-03-23 05:03:12,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:12,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:12,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-03-23 05:03:12,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-03-23 05:03:12,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:12,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:12,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-03-23 05:03:12,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-03-23 05:03:12,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:12,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:12,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-03-23 05:03:12,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-03-23 05:03:13,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:13,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:13,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-03-23 05:03:13,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-03-23 05:03:13,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:13,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:13,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-03-23 05:03:13,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-03-23 05:03:13,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:13,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:13,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-03-23 05:03:13,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-03-23 05:03:13,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:13,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:13,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-03-23 05:03:13,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-03-23 05:03:13,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:13,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:13,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-03-23 05:03:13,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-03-23 05:03:14,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:14,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:14,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-03-23 05:03:14,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-03-23 05:03:14,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:14,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:14,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-03-23 05:03:14,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-03-23 05:03:14,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:14,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:14,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-03-23 05:03:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-03-23 05:03:14,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:14,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:14,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-03-23 05:03:14,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-03-23 05:03:14,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:14,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:14,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-03-23 05:03:14,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-03-23 05:03:14,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:15,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:15,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-03-23 05:03:15,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-03-23 05:03:15,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.155: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-03-23 05:03:15,198 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:746] 172.23.105.155: No 'sanitizers.log.*' files found 2021-03-23 05:03:15,200 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.155 2021-03-23 05:03:15,209 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.211 with username: root, password: couchbase 2021-03-23 05:03:15,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:03:15,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /etc/ 2021-03-23 05:03:15,701 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:03:15,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /cygdrive/c/Windows 2021-03-23 05:03:15,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: uname -m 2021-03-23 05:03:15,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/cpuinfo 2021-03-23 05:03:15,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: df -Th 2021-03-23 05:03:15,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: cat /proc/meminfo 2021-03-23 05:03:15,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname 2021-03-23 05:03:15,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: hostname -d 2021-03-23 05:03:15,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-03-23 05:03:15,976 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:727] 172.23.105.211: No crash files found 2021-03-23 05:03:16,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-03-23 05:03:16,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:16,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:16,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-03-23 05:03:16,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-03-23 05:03:16,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:16,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:16,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-03-23 05:03:16,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-03-23 05:03:16,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:16,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:16,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-03-23 05:03:16,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-03-23 05:03:16,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:16,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:16,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-03-23 05:03:16,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-03-23 05:03:16,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:16,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:16,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-03-23 05:03:16,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-03-23 05:03:17,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:17,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:17,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-03-23 05:03:17,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-03-23 05:03:17,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:17,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:17,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-03-23 05:03:17,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-03-23 05:03:17,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:17,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:17,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-03-23 05:03:17,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-03-23 05:03:17,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:17,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:17,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-03-23 05:03:17,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-03-23 05:03:17,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:17,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:17,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt 2021-03-23 05:03:17,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt 2021-03-23 05:03:17,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000031.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000031.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:18,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000031.txt 2021-03-23 05:03:18,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000031.txt 2021-03-23 05:03:18,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000032.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000032.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:18,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000032.txt 2021-03-23 05:03:18,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000032.txt 2021-03-23 05:03:18,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000033.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000033.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:18,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000033.txt 2021-03-23 05:03:18,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000033.txt 2021-03-23 05:03:18,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000034.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000034.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:18,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000034.txt 2021-03-23 05:03:18,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000034.txt 2021-03-23 05:03:18,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000035.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000035.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:18,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000035.txt 2021-03-23 05:03:18,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000035.txt 2021-03-23 05:03:18,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000036.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:18,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000036.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:19,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000036.txt 2021-03-23 05:03:19,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000036.txt 2021-03-23 05:03:19,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:19,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:19,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt 2021-03-23 05:03:19,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt 2021-03-23 05:03:19,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:19,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:19,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt 2021-03-23 05:03:19,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt 2021-03-23 05:03:19,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000039.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:19,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000039.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:19,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000039.txt 2021-03-23 05:03:19,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000039.txt 2021-03-23 05:03:19,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000040.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:19,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000040.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:19,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000040.txt 2021-03-23 05:03:19,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000040.txt 2021-03-23 05:03:19,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.211: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-03-23 05:03:19,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:746] 172.23.105.211: No 'sanitizers.log.*' files found 2021-03-23 05:03:19,858 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.211 2021-03-23 05:03:19,865 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.212 with username: root, password: couchbase 2021-03-23 05:03:20,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:03:20,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: ls /etc/ 2021-03-23 05:03:20,415 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:03:20,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: ls /cygdrive/c/Windows 2021-03-23 05:03:20,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: uname -m 2021-03-23 05:03:20,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: cat /proc/cpuinfo 2021-03-23 05:03:20,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: df -Th 2021-03-23 05:03:20,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: cat /proc/meminfo 2021-03-23 05:03:20,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: hostname 2021-03-23 05:03:20,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: hostname -d 2021-03-23 05:03:20,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-03-23 05:03:20,709 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:727] 172.23.105.212: No crash files found 2021-03-23 05:03:20,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-03-23 05:03:20,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:20,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:20,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:20,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:21,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:21,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:21,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-03-23 05:03:21,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-03-23 05:03:21,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:21,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:21,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-03-23 05:03:21,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-03-23 05:03:21,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:21,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:21,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-03-23 05:03:21,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-03-23 05:03:21,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:21,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:21,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-03-23 05:03:21,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-03-23 05:03:21,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:21,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:21,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-03-23 05:03:21,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-03-23 05:03:22,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:22,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:22,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-03-23 05:03:22,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-03-23 05:03:22,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:22,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:22,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-03-23 05:03:22,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-03-23 05:03:22,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.212: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-03-23 05:03:22,447 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:746] 172.23.105.212: No 'sanitizers.log.*' files found 2021-03-23 05:03:22,448 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.212 2021-03-23 05:03:22,454 | infra | DEBUG | MainThread | [remote_util:connect:205] Connecting to 172.23.105.213 with username: root, password: couchbase 2021-03-23 05:03:22,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-03-23 05:03:22,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: ls /etc/ 2021-03-23 05:03:23,013 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3457] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-23 05:03:23,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: ls /cygdrive/c/Windows 2021-03-23 05:03:23,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: uname -m 2021-03-23 05:03:23,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: cat /proc/cpuinfo 2021-03-23 05:03:23,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: df -Th 2021-03-23 05:03:23,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: cat /proc/meminfo 2021-03-23 05:03:23,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: hostname 2021-03-23 05:03:23,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: hostname -d 2021-03-23 05:03:23,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-03-23 05:03:23,293 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:727] 172.23.105.213: No crash files found 2021-03-23 05:03:23,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-03-23 05:03:23,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' 2021-03-23 05:03:23,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' 2021-03-23 05:03:23,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:23,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-03-23 05:03:23,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3270] Running command on 172.23.105.213: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-03-23 05:03:23,553 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:746] 172.23.105.213: No 'sanitizers.log.*' files found 2021-03-23 05:03:23,555 | infra | DEBUG | MainThread | [remote_util:disconnect:218] Disconnecting ssh_client for 172.23.105.213 2021-03-23 05:03:23,579 | test | WARNING | MainThread | [basetestcase:tearDownEverything:466] CLEANUP WAS SKIPPED 2021-03-23 05:03:23,581 | infra | INFO | MainThread | [basetestcase:tearDownEverything:508] ========== tasks in thread pool ========== 2021-03-23 05:03:23,581 | infra | INFO | MainThread | [basetestcase:tearDownEverything:510] ========================================== During the test, Remote Connections: 21, Disconnections: 20 SDK Connections: 11, Disconnections: 11 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! ====================================================================== FAIL: test_sync_write_in_progress (epengine.collection_crud_negative.CollectionDurabilityTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/collection_crud_negative.py", line 501, in test_sync_write_in_progress self.validate_test_failure() File "pytests/basetestcase.py", line 627, in validate_test_failure self.fail(self.test_failure) AssertionError: Invalid retry reason for test_collections-100: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0x893037d30000000b","idempotent":false,"lastChannelId":"893037D30000000B/000000004AC48F00","lastDispatchedFrom":"172.23.106.205:54034","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":1421,"requestType":"InsertRequest","retried":0,"service":{"bucket":"JlW2lMteROclESYkprgr_%Af8lRXCvoWklQrL-28-142000","collection":"VJbuQNbBrT72XI1f15FXxd_ubU0kIllSrZkaELxbmrZ-XTu-XdCKHmAuUwQsULUbIWcuw2ABLPdx_bRFFj7JalAj275k4zsAO-X%HvjLiZJ2q60SoLJI8%MhlRypN5kgxhPhFHZo0jhRRnRDd3deaFuwbKKMuH5e0JrVpbeVLbeU9I5q8XfLHVNtKHI2paEB%ygI-28-144000","documentId":"test_collections-100","opaque":"0x1d2e","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":88,"totalMicros":310}} ---------------------------------------------------------------------- Ran 1 test in 98.631s FAILED (failures=1) summary so far suite epengine.collection_crud_negative.CollectionDurabilityTests , pass 0 , fail 1 failures so far... epengine.collection_crud_negative.CollectionDurabilityTests.test_sync_write_in_progress testrunner logs, diags and results are available under /data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Mar-23_05-01-43/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1m 47s 2 actionable tasks: 1 executed, 1 up-to-date [root@s83008 temp_centos7_4node_jython]#