[2022-06-02 13:02:38,862] - [eventing_base] [139727030187776] - INFO - ============== EventingBaseTest setup has started ============== [2022-06-02 13:02:38,863] - [eventing_base] [139727030187776] - INFO - Starting Test: test_expired_mutation [2022-06-02 13:02:38,864] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:02:38,965] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:02:39,243] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:39,502] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:39,507] - [rest_client] [139727030187776] - INFO - -->is_ns_server_running? [2022-06-02 13:02:39,519] - [rest_client] [139727030187776] - INFO - Node version in cluster 7.1.1-3105-enterprise [2022-06-02 13:02:39,519] - [basetestcase] [139727030187776] - INFO - ============== basetestcase setup was started for test #2 test_expired_mutation============== [2022-06-02 13:02:39,519] - [basetestcase] [139727030187776] - INFO - Building docker image with java sdk client [2022-06-02 13:02:45,418] - [basetestcase] [139727030187776] - INFO - initializing cluster [2022-06-02 13:02:45,877] - [task] [139728036820736] - INFO - server: ip:172.23.121.143 port:8091 ssh_username:root, nodes/self [2022-06-02 13:02:45,881] - [task] [139728036820736] - INFO - {'uptime': '3774', 'memoryTotal': 12294119424, 'memoryFree': 11165216768, 'mcdMemoryReserved': 9379, 'mcdMemoryAllocated': 9379, 'status': 'healthy', 'hostname': '172.23.121.143:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3105-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2400, 'memcached': 11210, 'id': 'ns_1@172.23.121.143', 'ip': '172.23.121.143', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 11724, 'curr_items': 0} [2022-06-02 13:02:45,885] - [rest_client] [139728036820736] - INFO - pools/default params : memoryQuota=6283 [2022-06-02 13:02:45,889] - [rest_client] [139728036820736] - INFO - --> init_node_services(Administrator,password,172.23.121.143,8091,['kv']) [2022-06-02 13:02:45,890] - [rest_client] [139728036820736] - INFO - /node/controller/setupServices params on 172.23.121.143: 8091:hostname=172.23.121.143&user=Administrator&password=password&services=kv [2022-06-02 13:02:45,892] - [rest_client] [139728036820736] - ERROR - POST http://172.23.121.143:8091//node/controller/setupServices body: hostname=172.23.121.143&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-06-02 13:02:45,892] - [rest_client] [139728036820736] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-06-02 13:02:45,892] - [rest_client] [139728036820736] - INFO - --> in init_cluster...Administrator,password,8091 [2022-06-02 13:02:45,893] - [rest_client] [139728036820736] - INFO - settings/web params on 172.23.121.143:8091:port=8091&username=Administrator&password=password [2022-06-02 13:02:45,941] - [rest_client] [139728036820736] - INFO - --> status:True [2022-06-02 13:02:45,943] - [remote_util] [139728036820736] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:02:46,047] - [remote_util] [139728036820736] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:02:46,311] - [remote_util] [139728036820736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:46,565] - [remote_util] [139728036820736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:46,566] - [remote_util] [139728036820736] - INFO - running command.raw on 172.23.121.143: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-06-02 13:02:46,625] - [remote_util] [139728036820736] - INFO - command executed successfully with root [2022-06-02 13:02:46,626] - [remote_util] [139728036820736] - INFO - ['ok'] [2022-06-02 13:02:46,629] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.121.143:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:46,636] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.121.143:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:46,641] - [rest_client] [139728036820736] - INFO - settings/indexes params : storageMode=plasma [2022-06-02 13:02:46,657] - [task] [139728036820736] - INFO - server: ip:172.23.121.79 port:8091 ssh_username:root, nodes/self [2022-06-02 13:02:46,661] - [task] [139728036820736] - INFO - {'uptime': '3777', 'memoryTotal': 12294160384, 'memoryFree': 11120869376, 'mcdMemoryReserved': 9379, 'mcdMemoryAllocated': 9379, 'status': 'healthy', 'hostname': '172.23.121.79:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3105-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 6368, 'memcached': 11210, 'id': 'ns_1@172.23.121.79', 'ip': '172.23.121.79', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 11724, 'curr_items': 0} [2022-06-02 13:02:46,664] - [rest_client] [139728036820736] - ERROR - GET http://172.23.121.79:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-06-02 13:02:46,664] - [rest_client] [139728036820736] - INFO - pools/default params : memoryQuota=6283 [2022-06-02 13:02:46,668] - [rest_client] [139728036820736] - INFO - --> in init_cluster...Administrator,password,8091 [2022-06-02 13:02:46,668] - [rest_client] [139728036820736] - INFO - settings/web params on 172.23.121.79:8091:port=8091&username=Administrator&password=password [2022-06-02 13:02:46,724] - [rest_client] [139728036820736] - INFO - --> status:True [2022-06-02 13:02:46,726] - [remote_util] [139728036820736] - INFO - SSH Connecting to 172.23.121.79 with username:root, attempt#1 of 5 [2022-06-02 13:02:46,830] - [remote_util] [139728036820736] - INFO - SSH Connected to 172.23.121.79 as root [2022-06-02 13:02:47,103] - [remote_util] [139728036820736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:47,352] - [remote_util] [139728036820736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:47,353] - [remote_util] [139728036820736] - INFO - running command.raw on 172.23.121.79: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-06-02 13:02:47,412] - [remote_util] [139728036820736] - INFO - command executed successfully with root [2022-06-02 13:02:47,412] - [remote_util] [139728036820736] - INFO - ['ok'] [2022-06-02 13:02:47,415] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.121.79:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:47,418] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.121.79:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:47,423] - [rest_client] [139728036820736] - INFO - settings/indexes params : storageMode=plasma [2022-06-02 13:02:47,436] - [task] [139728036820736] - INFO - server: ip:172.23.122.221 port:8091 ssh_username:root, nodes/self [2022-06-02 13:02:47,439] - [task] [139728036820736] - INFO - {'uptime': '3771', 'memoryTotal': 12294160384, 'memoryFree': 11141230592, 'mcdMemoryReserved': 9379, 'mcdMemoryAllocated': 9379, 'status': 'healthy', 'hostname': '172.23.122.221:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3105-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 6368, 'memcached': 11210, 'id': 'ns_1@172.23.122.221', 'ip': '172.23.122.221', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 11724, 'curr_items': 0} [2022-06-02 13:02:47,441] - [rest_client] [139728036820736] - ERROR - GET http://172.23.122.221:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-06-02 13:02:47,441] - [rest_client] [139728036820736] - INFO - pools/default params : memoryQuota=6283 [2022-06-02 13:02:47,445] - [rest_client] [139728036820736] - INFO - --> in init_cluster...Administrator,password,8091 [2022-06-02 13:02:47,445] - [rest_client] [139728036820736] - INFO - settings/web params on 172.23.122.221:8091:port=8091&username=Administrator&password=password [2022-06-02 13:02:47,499] - [rest_client] [139728036820736] - INFO - --> status:True [2022-06-02 13:02:47,501] - [remote_util] [139728036820736] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:02:47,603] - [remote_util] [139728036820736] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:02:47,880] - [remote_util] [139728036820736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:48,141] - [remote_util] [139728036820736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:48,142] - [remote_util] [139728036820736] - INFO - running command.raw on 172.23.122.221: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-06-02 13:02:48,202] - [remote_util] [139728036820736] - INFO - command executed successfully with root [2022-06-02 13:02:48,202] - [remote_util] [139728036820736] - INFO - ['ok'] [2022-06-02 13:02:48,205] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.122.221:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:48,207] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.122.221:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:48,210] - [rest_client] [139728036820736] - INFO - settings/indexes params : storageMode=plasma [2022-06-02 13:02:48,222] - [task] [139728036820736] - INFO - server: ip:172.23.122.208 port:8091 ssh_username:root, nodes/self [2022-06-02 13:02:48,225] - [task] [139728036820736] - INFO - {'uptime': '3745', 'memoryTotal': 12294164480, 'memoryFree': 11457892352, 'mcdMemoryReserved': 9379, 'mcdMemoryAllocated': 9379, 'status': 'healthy', 'hostname': '172.23.122.208:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3105-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 6368, 'memcached': 11210, 'id': 'ns_1@172.23.122.208', 'ip': '172.23.122.208', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 11724, 'curr_items': 0} [2022-06-02 13:02:48,226] - [rest_client] [139728036820736] - ERROR - GET http://172.23.122.208:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-06-02 13:02:48,227] - [rest_client] [139728036820736] - INFO - pools/default params : memoryQuota=6283 [2022-06-02 13:02:48,230] - [rest_client] [139728036820736] - INFO - --> in init_cluster...Administrator,password,8091 [2022-06-02 13:02:48,230] - [rest_client] [139728036820736] - INFO - settings/web params on 172.23.122.208:8091:port=8091&username=Administrator&password=password [2022-06-02 13:02:48,287] - [rest_client] [139728036820736] - INFO - --> status:True [2022-06-02 13:02:48,289] - [remote_util] [139728036820736] - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 [2022-06-02 13:02:48,410] - [remote_util] [139728036820736] - INFO - SSH Connected to 172.23.122.208 as root [2022-06-02 13:02:48,718] - [remote_util] [139728036820736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:48,986] - [remote_util] [139728036820736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:48,987] - [remote_util] [139728036820736] - INFO - running command.raw on 172.23.122.208: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-06-02 13:02:49,045] - [remote_util] [139728036820736] - INFO - command executed successfully with root [2022-06-02 13:02:49,046] - [remote_util] [139728036820736] - INFO - ['ok'] [2022-06-02 13:02:49,049] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.122.208:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:49,052] - [rest_client] [139728036820736] - INFO - /diag/eval status on 172.23.122.208:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-06-02 13:02:49,060] - [rest_client] [139728036820736] - INFO - settings/indexes params : storageMode=plasma [2022-06-02 13:02:49,073] - [basetestcase] [139727030187776] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.121.143 **** [2022-06-02 13:02:49,161] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. ... [2022-06-02 13:02:54,166] - [basetestcase] [139727030187776] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-06-02 13:02:54,178] - [basetestcase] [139727030187776] - INFO - done initializing cluster [2022-06-02 13:02:54,179] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:02:54,281] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:02:54,555] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:54,819] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:54,967] - [remote_util] [139727030187776] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.121.143:8091 -u Administrator -p password --enable [2022-06-02 13:02:55,318] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.79 with username:root, attempt#1 of 5 [2022-06-02 13:02:55,425] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.79 as root [2022-06-02 13:02:55,714] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:55,976] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:56,111] - [remote_util] [139727030187776] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.121.79:8091 -u Administrator -p password --enable [2022-06-02 13:02:56,467] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:02:56,576] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:02:56,844] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:57,100] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:57,215] - [remote_util] [139727030187776] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.122.221:8091 -u Administrator -p password --enable [2022-06-02 13:02:57,559] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 [2022-06-02 13:02:57,669] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.208 as root [2022-06-02 13:02:57,963] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:02:58,233] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:02:58,375] - [remote_util] [139727030187776] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.122.208:8091 -u Administrator -p password --enable [2022-06-02 13:02:59,084] - [task] [139728036820736] - INFO - adding node 172.23.121.79:8091 to cluster [2022-06-02 13:02:59,093] - [rest_client] [139728036820736] - INFO - adding remote node @172.23.121.79:18091 to this cluster @172.23.121.143:8091 [2022-06-02 13:03:09,108] - [rest_client] [139728036820736] - INFO - rebalance progress took 10.01 seconds [2022-06-02 13:03:09,108] - [rest_client] [139728036820736] - INFO - sleep for 10 seconds after rebalance... [2022-06-02 13:03:22,224] - [task] [139728036820736] - INFO - adding node 172.23.122.221:8091 to cluster [2022-06-02 13:03:22,231] - [rest_client] [139728036820736] - INFO - adding remote node @172.23.122.221:18091 to this cluster @172.23.121.143:8091 [2022-06-02 13:03:32,245] - [rest_client] [139728036820736] - INFO - rebalance progress took 10.01 seconds [2022-06-02 13:03:32,245] - [rest_client] [139728036820736] - INFO - sleep for 10 seconds after rebalance... [2022-06-02 13:03:45,852] - [task] [139728036820736] - INFO - adding node 172.23.122.208:8091 to cluster [2022-06-02 13:03:45,861] - [rest_client] [139728036820736] - INFO - adding remote node @172.23.122.208:18091 to this cluster @172.23.121.143:8091 [2022-06-02 13:03:55,877] - [rest_client] [139728036820736] - INFO - rebalance progress took 10.02 seconds [2022-06-02 13:03:55,877] - [rest_client] [139728036820736] - INFO - sleep for 10 seconds after rebalance... [2022-06-02 13:04:09,396] - [rest_client] [139728036820736] - INFO - Node 172.23.121.79 not part of cluster inactiveAdded [2022-06-02 13:04:09,396] - [rest_client] [139728036820736] - INFO - Node 172.23.122.208 not part of cluster inactiveAdded [2022-06-02 13:04:09,396] - [rest_client] [139728036820736] - INFO - Node 172.23.122.221 not part of cluster inactiveAdded [2022-06-02 13:04:09,402] - [rest_client] [139728036820736] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.121.143,ns_1@172.23.121.79,ns_1@172.23.122.208,ns_1@172.23.122.221', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-06-02 13:04:09,451] - [rest_client] [139728036820736] - INFO - rebalance operation started [2022-06-02 13:04:19,464] - [task] [139728036820736] - INFO - Rebalance - status: none, progress: 100.00% [2022-06-02 13:04:19,470] - [task] [139728036820736] - INFO - rebalancing was completed with progress: 100% in 10.018725156784058 sec [2022-06-02 13:04:19,472] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:04:19,576] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:04:19,842] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:20,095] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:20,097] - [remote_util] [139727030187776] - INFO - running command.raw on 172.23.121.143: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-06-02 13:04:20,153] - [remote_util] [139727030187776] - INFO - command executed successfully with root [2022-06-02 13:04:20,153] - [remote_util] [139727030187776] - INFO - ['ok'] [2022-06-02 13:04:20,153] - [basetestcase] [139727030187776] - INFO - Enabled diag/eval for non-local hosts from 172.23.121.143 [2022-06-02 13:04:20,154] - [basetestcase] [139727030187776] - INFO - ============== basetestcase setup was finished for test #2 test_expired_mutation ============== [2022-06-02 13:04:20,162] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:04:20,261] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:04:20,515] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:20,768] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:20,770] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.79 with username:root, attempt#1 of 5 [2022-06-02 13:04:20,870] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.79 as root [2022-06-02 13:04:21,136] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:21,391] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:21,393] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:04:21,493] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:04:21,756] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:22,006] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:22,009] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 [2022-06-02 13:04:22,122] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.208 as root [2022-06-02 13:04:22,413] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:22,671] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - ------- Cluster statistics ------- [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - 172.23.121.143:8091 => {'services': ['kv'], 'cpu_utilization': 6.77084057251634, 'mem_free': 11174887424, 'mem_total': 12294119424, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - 172.23.121.79:8091 => {'services': ['kv'], 'cpu_utilization': 1.079867837410158, 'mem_free': 11103768576, 'mem_total': 12294160384, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - 172.23.122.208:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.3465597025515795, 'mem_free': 11374264320, 'mem_total': 12294164480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - 172.23.122.221:8091 => {'services': ['eventing'], 'cpu_utilization': 0.874636151500951, 'mem_free': 11137974272, 'mem_total': 12294160384, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - --- End of cluster statistics --- [2022-06-02 13:04:30,457] - [basetestcase] [139727030187776] - INFO - Time to execute basesetup : 111.59476208686829 [2022-06-02 13:04:30,459] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:04:30,561] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:04:30,824] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:04:31,074] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:04:38,968] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:04:38,975] - [basetestcase] [139727030187776] - INFO - list of eventing nodes in cluster: [ip:172.23.122.221 port:8091 ssh_username:root] [2022-06-02 13:04:38,980] - [rest_client] [139727030187776] - INFO - settings/indexes params : storageMode=plasma [2022-06-02 13:04:38,985] - [eventing_base] [139727030187776] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2022-06-02 13:04:38,985] - [rest_client] [139727030187776] - INFO - pools/default params : memoryQuota=330 [2022-06-02 13:04:38,990] - [rest_client] [139727030187776] - INFO - pools/default params : indexMemoryQuota=256 [2022-06-02 13:04:39,000] - [rest_client] [139727030187776] - INFO - pools/default params : eventingMemoryQuota=512 [2022-06-02 13:04:39,016] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:04:39,026] - [eventing_base] [139727030187776] - INFO - ============== EventingBaseTest setup has completed ============== [2022-06-02 13:04:39,026] - [rest_client] [139727030187776] - INFO - pools/default params : memoryQuota=2400 [2022-06-02 13:04:39,032] - [eventing_sanity] [139727030187776] - INFO - 512 [2022-06-02 13:04:39,525] - [rest_client] [139728036820736] - INFO - http://172.23.121.143:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=512&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=fullEviction&compressionMode=passive&storageBackend=magma [2022-06-02 13:04:39,552] - [rest_client] [139728036820736] - INFO - 0.03 seconds to create bucket src_bucket [2022-06-02 13:04:39,552] - [bucket_helper] [139728036820736] - INFO - waiting for memcached bucket : src_bucket in 172.23.121.143 to accept set ops [2022-06-02 13:04:40,175] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 src_bucket [2022-06-02 13:04:40,257] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 src_bucket [2022-06-02 13:04:42,017] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 src_bucket [2022-06-02 13:04:42,084] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 src_bucket [2022-06-02 13:04:42,596] - [task] [139728036820736] - INFO - bucket 'src_bucket' was created with per node RAM quota: 512 [2022-06-02 13:04:43,617] - [rest_client] [139728036820736] - INFO - http://172.23.121.143:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=512&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=fullEviction&compressionMode=passive&storageBackend=magma [2022-06-02 13:04:43,645] - [rest_client] [139728036820736] - INFO - 0.03 seconds to create bucket dst_bucket [2022-06-02 13:04:43,645] - [bucket_helper] [139728036820736] - INFO - waiting for memcached bucket : dst_bucket in 172.23.121.143 to accept set ops [2022-06-02 13:04:44,291] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 dst_bucket [2022-06-02 13:04:44,390] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 dst_bucket [2022-06-02 13:04:44,490] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 dst_bucket [2022-06-02 13:04:44,547] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 dst_bucket [2022-06-02 13:04:45,916] - [task] [139728036820736] - INFO - bucket 'dst_bucket' was created with per node RAM quota: 512 [2022-06-02 13:04:46,940] - [rest_client] [139728036820736] - INFO - http://172.23.121.143:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=512&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=fullEviction&compressionMode=passive&storageBackend=magma [2022-06-02 13:04:46,969] - [rest_client] [139728036820736] - INFO - 0.03 seconds to create bucket metadata [2022-06-02 13:04:46,970] - [bucket_helper] [139728036820736] - INFO - waiting for memcached bucket : metadata in 172.23.121.143 to accept set ops [2022-06-02 13:04:47,616] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 metadata [2022-06-02 13:04:47,705] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 metadata [2022-06-02 13:04:47,840] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.143:11210 metadata [2022-06-02 13:04:47,920] - [data_helper] [139728036820736] - INFO - creating direct client 172.23.121.79:11210 metadata [2022-06-02 13:04:49,193] - [task] [139728036820736] - INFO - bucket 'metadata' was created with per node RAM quota: 512 [2022-06-02 13:04:49,859] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/src_bucket/scopes with params: name=src_bucket [2022-06-02 13:04:49,859] - [rest_client] [139727030187776] - INFO - Scope created src_bucket->src_bucket {'uid': '1'} [2022-06-02 13:04:49,912] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/src_bucket/scopes/src_bucket/collections with params: name=src_bucket [2022-06-02 13:04:49,913] - [rest_client] [139727030187776] - INFO - Collection created src_bucket->src_bucket->src_bucket manifest:{'uid': '2'} [2022-06-02 13:04:49,966] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/metadata/scopes with params: name=metadata [2022-06-02 13:04:49,967] - [rest_client] [139727030187776] - INFO - Scope created metadata->metadata {'uid': '1'} [2022-06-02 13:04:50,029] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/metadata/scopes/metadata/collections with params: name=metadata [2022-06-02 13:04:50,029] - [rest_client] [139727030187776] - INFO - Collection created metadata->metadata->metadata manifest:{'uid': '2'} [2022-06-02 13:04:50,077] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/dst_bucket/scopes with params: name=dst_bucket [2022-06-02 13:04:50,077] - [rest_client] [139727030187776] - INFO - Scope created dst_bucket->dst_bucket {'uid': '1'} [2022-06-02 13:04:50,135] - [rest_client] [139727030187776] - INFO - http://172.23.121.143:8091/pools/default/buckets/dst_bucket/scopes/dst_bucket/collections with params: name=dst_bucket [2022-06-02 13:04:50,135] - [rest_client] [139727030187776] - INFO - Collection created dst_bucket->dst_bucket->dst_bucket manifest:{'uid': '2'} [2022-06-02 13:04:50,142] - [tuq_helper] [139727030187776] - INFO - RUN QUERY create primary index on src_bucket [2022-06-02 13:04:50,148] - [rest_client] [139727030187776] - INFO - query params : statement=create+primary+index+on+src_bucket [2022-06-02 13:04:52,360] - [tuq_helper] [139727030187776] - INFO - TOTAL ELAPSED TIME: 2.210060181s [2022-06-02 13:04:52,366] - [tuq_helper] [139727030187776] - INFO - RUN QUERY create primary index on dst_bucket [2022-06-02 13:04:52,371] - [rest_client] [139727030187776] - INFO - query params : statement=create+primary+index+on+dst_bucket [2022-06-02 13:04:54,433] - [tuq_helper] [139727030187776] - INFO - TOTAL ELAPSED TIME: 2.059398417s [2022-06-02 13:04:54,440] - [tuq_helper] [139727030187776] - INFO - RUN QUERY create primary index on metadata [2022-06-02 13:04:54,446] - [rest_client] [139727030187776] - INFO - query params : statement=create+primary+index+on+metadata [2022-06-02 13:04:56,693] - [tuq_helper] [139727030187776] - INFO - TOTAL ELAPSED TIME: 2.243358106s [2022-06-02 13:04:56,694] - [basetestcase] [139727030187776] - INFO - Number of batches : 13 [2022-06-02 13:04:56,694] - [basetestcase] [139727030187776] - INFO - sleep for 10 secs. ... [2022-06-02 13:05:06,705] - [basetestcase] [139727030187776] - INFO - done [2022-06-02 13:05:06,733] - [data_helper] [139727030187776] - INFO - dict:{'ip': '172.23.121.143', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2022-06-02 13:05:06,733] - [data_helper] [139727030187776] - INFO - creating direct client 172.23.121.143:11210 src_bucket [2022-06-02 13:05:06,775] - [cluster_helper] [139727030187776] - INFO - Setting flush param on server {'ip': '172.23.121.143', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 1 on src_bucket [2022-06-02 13:05:06,775] - [mc_bin_client] [139727030187776] - INFO - setting param: exp_pager_stime 1 [2022-06-02 13:05:06,776] - [cluster_helper] [139727030187776] - INFO - Setting flush param on server {'ip': '172.23.121.143', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 1, result: (2171049995, 0, b'') [2022-06-02 13:05:06,792] - [data_helper] [139727030187776] - INFO - dict:{'ip': '172.23.121.79', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2022-06-02 13:05:06,792] - [data_helper] [139727030187776] - INFO - creating direct client 172.23.121.79:11210 src_bucket [2022-06-02 13:05:06,841] - [cluster_helper] [139727030187776] - INFO - Setting flush param on server {'ip': '172.23.121.79', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 1 on src_bucket [2022-06-02 13:05:06,841] - [mc_bin_client] [139727030187776] - INFO - setting param: exp_pager_stime 1 [2022-06-02 13:05:06,842] - [cluster_helper] [139727030187776] - INFO - Setting flush param on server {'ip': '172.23.121.79', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 1, result: (1574334605, 0, b'') [2022-06-02 13:05:07,214] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:05:07,901] - [eventing_base] [139727030187776] - INFO - saving function b'{\n "code": 0,\n "info": "Function: src_bucket/src_bucket/Function_995760096_test_expired_mutation stored in temp store"\n}' [2022-06-02 13:05:07,901] - [eventing_base] [139727030187776] - INFO - Deploying the following handler code : Function_995760096_test_expired_mutation with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'scope_name': 'dst_bucket', 'collection_name': 'dst_bucket', 'access': 'rw'}], 'metadata_bucket': 'metadata', 'metadata_scope': 'metadata', 'metadata_collection': 'metadata', 'source_bucket': 'src_bucket', 'source_scope': 'src_bucket', 'source_collection': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.6.2'} [2022-06-02 13:05:07,902] - [eventing_base] [139727030187776] - INFO - function OnUpdate(doc, meta) { log('docId', meta.id); dst_bucket[meta.id]=doc; } function OnDelete(meta, options) { if(options.expired){ log('expired doc', meta.id); delete dst_bucket[meta.id]; } } [2022-06-02 13:05:08,959] - [eventing_base] [139727030187776] - INFO - deploy Application : b'{\n "code": 0,\n "info": "Function: src_bucket/src_bucket/Function_995760096_test_expired_mutation is deploying"\n}' [2022-06-02 13:05:08,960] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to deployed... ... [2022-06-02 13:05:13,972] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to deployed... ... [2022-06-02 13:05:18,984] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to deployed... ... [2022-06-02 13:05:23,996] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to deployed... ... [2022-06-02 13:05:29,020] - [basetestcase] [139727030187776] - INFO - list of eventing nodes in cluster: [ip:172.23.122.221 port:8091 ssh_username:root] [2022-06-02 13:05:29,027] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:05:29,032] - [tuq_helper] [139727030187776] - INFO - RUN QUERY create primary index on src_bucket.src_bucket.src_bucket [2022-06-02 13:05:29,038] - [rest_client] [139727030187776] - INFO - query params : statement=create+primary+index+on+src_bucket.src_bucket.src_bucket [2022-06-02 13:05:55,122] - [tuq_helper] [139727030187776] - INFO - TOTAL ELAPSED TIME: 26.080690227s [2022-06-02 13:05:55,133] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:05:55,140] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:05:55,158] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 100000 Expected : 0 ... [2022-06-02 13:06:05,029] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 100001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:06:25,218] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:06:25,225] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:06:25,246] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:06:25,252] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:06:25,256] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 131837 Expected : 0 ... [2022-06-02 13:06:55,272] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:06:55,278] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:06:55,297] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:06:55,304] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:06:55,307] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 185778 Expected : 0 ... [2022-06-02 13:07:02,587] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 200001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:07:25,353] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:07:25,359] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:07:25,379] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:07:25,384] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:07:25,388] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 173873 Expected : 0 ... [2022-06-02 13:07:55,431] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:07:55,438] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:07:55,461] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:07:55,467] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:07:55,471] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 177244 Expected : 0 ... [2022-06-02 13:07:58,613] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 300001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:08:25,516] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:08:25,521] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:08:25,540] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:08:25,544] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:08:25,548] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 173104 Expected : 0 ... [2022-06-02 13:08:55,239] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 400001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:08:55,593] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:08:55,599] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:08:55,617] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:08:55,621] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:08:55,625] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 187274 Expected : 0 ... [2022-06-02 13:09:25,661] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:09:25,667] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:09:25,688] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:09:25,693] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:09:25,697] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 179501 Expected : 0 ... [2022-06-02 13:09:51,335] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 500001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:09:55,739] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:09:55,744] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:09:55,764] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:09:55,769] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:09:55,773] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 181861 Expected : 0 ... [2022-06-02 13:10:25,817] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:10:25,824] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:10:25,844] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:10:25,849] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:10:25,853] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 180137 Expected : 0 ... [2022-06-02 13:10:46,494] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 600001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:10:55,898] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:10:55,903] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:10:55,920] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:10:55,926] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:10:55,930] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 177001 Expected : 0 ... [2022-06-02 13:11:25,975] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:11:25,980] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:11:26,000] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:11:26,004] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:11:26,007] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 177449 Expected : 0 ... [2022-06-02 13:11:44,083] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 700001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:11:56,052] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:11:56,058] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:11:56,076] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:11:56,081] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:11:56,084] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 178987 Expected : 0 ... [2022-06-02 13:12:26,128] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:12:26,133] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:12:26,149] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:12:26,154] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:12:26,157] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 185509 Expected : 0 ... [2022-06-02 13:12:40,112] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 800001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:12:56,183] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:12:56,189] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:12:56,210] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:12:56,215] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:12:56,220] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 191936 Expected : 0 ... [2022-06-02 13:13:26,244] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:13:26,251] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:13:26,267] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:13:26,271] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:13:26,274] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 198742 Expected : 0 ... [2022-06-02 13:13:37,125] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 900001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:13:56,321] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:13:56,328] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:13:56,347] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:13:56,351] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:13:56,355] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 202937 Expected : 0 ... [2022-06-02 13:14:27,401] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:14:27,406] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:14:27,427] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:14:27,432] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:14:27,436] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 210379 Expected : 0 ... [2022-06-02 13:14:34,252] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1000001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:14:57,476] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:14:57,483] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:14:57,501] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:14:57,506] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:14:57,509] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 218150 Expected : 0 ... [2022-06-02 13:15:27,527] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:15:27,534] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:15:27,553] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:15:27,558] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:15:27,562] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 223700 Expected : 0 ... [2022-06-02 13:15:38,840] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 100000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1100001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:15:57,608] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:15:57,615] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:15:57,634] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:15:57,639] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:15:57,643] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 222458 Expected : 0 ... [2022-06-02 13:16:27,688] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:16:27,693] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:16:27,712] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:16:27,719] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:16:27,722] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 247792 Expected : 0 ... [2022-06-02 13:16:36,528] - [task] [139728036820736] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.121.143 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 50000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1200001 -dpx doc_ -dt Person -de 100 -ds 512 -ac False -st 0 -en 1249999 -o False -sd False --secure False [2022-06-02 13:16:57,935] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:16:57,942] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:16:57,962] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:16:57,968] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:16:57,971] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 259725 Expected : 0 ... [2022-06-02 13:17:28,013] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:17:28,018] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:17:28,036] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:17:28,041] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:17:28,044] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 264651 Expected : 0 ... [2022-06-02 13:17:58,086] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:17:58,091] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:17:58,107] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:17:58,111] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:17:58,115] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 257825 Expected : 0 ... [2022-06-02 13:18:29,017] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:18:29,022] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:18:29,038] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:18:29,044] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:18:29,048] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 246324 Expected : 0 ... [2022-06-02 13:18:59,087] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:18:59,093] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:18:59,113] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:18:59,119] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:18:59,122] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 220127 Expected : 0 ... [2022-06-02 13:19:29,157] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:19:29,164] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:19:29,183] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:19:29,188] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:19:29,191] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 211226 Expected : 0 ... [2022-06-02 13:19:59,660] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:20:00,133] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:20:00,149] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:20:00,154] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:20:00,157] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 180184 Expected : 0 ... [2022-06-02 13:20:30,199] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:20:30,205] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:20:30,221] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:20:30,226] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:20:30,230] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 137483 Expected : 0 ... [2022-06-02 13:21:00,262] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:21:00,268] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:21:00,287] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:21:00,291] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:21:00,295] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 133997 Expected : 0 ... [2022-06-02 13:21:30,338] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:21:30,345] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:21:30,362] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:21:30,367] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:21:30,370] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 133765 Expected : 0 ... [2022-06-02 13:22:00,386] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:22:00,392] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:22:00,410] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:22:00,414] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:22:00,418] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 116255 Expected : 0 ... [2022-06-02 13:22:30,459] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:22:30,465] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:22:30,483] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:22:30,488] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:22:30,491] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 97610 Expected : 0 ... [2022-06-02 13:23:00,504] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:23:00,510] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:23:00,526] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:23:00,530] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:23:00,533] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:23:30,550] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:23:30,556] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:23:30,573] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:23:30,576] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:23:30,579] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:24:00,625] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:24:00,631] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:24:00,649] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:24:00,655] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:24:00,658] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:24:30,673] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:24:30,678] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:24:30,694] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:24:30,698] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:24:30,701] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:25:00,745] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:25:00,751] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:25:00,767] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:25:00,772] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:25:00,775] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:25:30,819] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:25:30,825] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:25:30,841] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:25:30,846] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:25:30,849] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:26:00,869] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:26:00,875] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:26:00,892] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:26:00,896] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:26:00,899] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:26:30,924] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:26:30,929] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:26:30,946] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:26:30,950] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:26:30,954] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:27:00,995] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:27:01,002] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:27:01,021] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:27:01,026] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:27:01,030] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:27:31,067] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:27:31,073] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:27:31,090] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:27:31,095] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:27:31,099] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:28:01,140] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:28:01,146] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:28:01,164] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:28:01,169] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:28:01,173] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:28:31,196] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:28:31,202] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:28:31,217] - [eventing_base] [139727030187776] - INFO - Compacting bucket : dst_bucket [2022-06-02 13:28:31,220] - [rest_client] [139727030187776] - INFO - bucket compaction successful [2022-06-02 13:28:31,221] - [eventing_base] [139727030187776] - INFO - Compacting bucket : metadata [2022-06-02 13:28:31,224] - [rest_client] [139727030187776] - INFO - bucket compaction successful [2022-06-02 13:28:31,224] - [eventing_base] [139727030187776] - INFO - Compacting bucket : src_bucket [2022-06-02 13:28:31,227] - [rest_client] [139727030187776] - INFO - bucket compaction successful [2022-06-02 13:28:31,234] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:28:31,239] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:28:31,243] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 94742 Expected : 0 ... [2022-06-02 13:29:01,271] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:29:01,277] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:29:01,295] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:29:01,300] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:29:01,303] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 91195 Expected : 0 ... [2022-06-02 13:29:31,344] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:29:31,350] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:29:31,368] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:29:31,372] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:29:31,375] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 88169 Expected : 0 ... [2022-06-02 13:30:01,414] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:30:01,419] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:30:01,434] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:30:01,437] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:30:01,440] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 84470 Expected : 0 ... [2022-06-02 13:30:31,484] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:30:31,492] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:30:31,512] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:30:31,518] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:30:31,521] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 80549 Expected : 0 ... [2022-06-02 13:31:01,564] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:31:01,569] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:31:01,585] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:31:01,588] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:31:01,591] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 77217 Expected : 0 ... [2022-06-02 13:31:31,637] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:31:31,643] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:31:31,660] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:31:31,664] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:31:31,667] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 73948 Expected : 0 ... [2022-06-02 13:32:01,704] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:32:01,712] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:32:01,731] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:32:01,739] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:32:01,742] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 62845 Expected : 0 ... [2022-06-02 13:32:31,762] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:32:31,769] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:32:31,790] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:32:31,796] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:32:31,799] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 47773 Expected : 0 ... [2022-06-02 13:33:01,840] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:33:01,846] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:33:01,865] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:33:01,870] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:33:01,874] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 38406 Expected : 0 ... [2022-06-02 13:33:31,918] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:33:31,923] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:33:31,940] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:33:31,945] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:33:31,948] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 32474 Expected : 0 ... [2022-06-02 13:34:01,984] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:34:01,990] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:34:02,006] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:34:02,012] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:34:02,016] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 25767 Expected : 0 ... [2022-06-02 13:34:32,059] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:34:32,065] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:34:32,081] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:34:32,087] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:34:32,090] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 20874 Expected : 0 ... [2022-06-02 13:35:02,133] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:35:02,141] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:35:02,159] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:35:02,163] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:35:02,167] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 17887 Expected : 0 ... [2022-06-02 13:35:32,209] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:35:32,214] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:35:32,230] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:35:32,235] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:35:32,238] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 12306 Expected : 0 ... [2022-06-02 13:36:02,256] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:36:02,262] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:36:02,279] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:36:02,283] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:36:02,286] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 8256 Expected : 0 ... [2022-06-02 13:36:32,326] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:36:32,332] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:36:32,347] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:36:32,352] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:36:32,355] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 5214 Expected : 0 ... [2022-06-02 13:37:02,375] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:37:02,380] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:37:02,397] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:37:02,401] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:37:02,404] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 5214 Expected : 0 ... [2022-06-02 13:37:32,431] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:37:32,437] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:37:32,455] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:37:32,461] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:37:32,464] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 5214 Expected : 0 ... [2022-06-02 13:38:02,480] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:38:02,486] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:38:02,504] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:38:02,509] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:38:02,513] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 4393 Expected : 0 ... [2022-06-02 13:38:32,553] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:38:32,559] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:38:32,576] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:38:32,582] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:38:32,585] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 3406 Expected : 0 ... [2022-06-02 13:39:02,604] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:39:02,610] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:39:02,626] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:39:02,631] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:39:02,635] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 3406 Expected : 0 ... [2022-06-02 13:39:32,676] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:39:32,682] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:39:32,698] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:39:32,702] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:39:32,704] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 3406 Expected : 0 ... [2022-06-02 13:40:02,747] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:40:02,753] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:40:02,772] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:40:02,776] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:40:02,779] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 3406 Expected : 0 ... [2022-06-02 13:40:32,822] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:40:32,829] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:40:32,847] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:40:32,852] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:40:32,855] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 3406 Expected : 0 ... [2022-06-02 13:41:02,892] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:41:02,900] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:41:02,917] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:41:02,922] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:41:02,926] - [basetestcase] [139727030187776] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.src_bucket.src_bucket to complete bucket operations... Current : 1040 Expected : 0 ... [2022-06-02 13:41:32,969] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:41:32,976] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:41:32,996] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:41:33,000] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.src_bucket.src_bucket [2022-06-02 13:41:33,002] - [eventing_base] [139727030187776] - INFO - Final docs count... Current : 0 Expected : 0 [2022-06-02 13:41:33,009] - [basetestcase] [139727030187776] - INFO - list of eventing nodes in cluster: [ip:172.23.122.221 port:8091 ssh_username:root] [2022-06-02 13:41:33,045] - [eventing_base] [139727030187776] - INFO - Undeploy Application : Function_995760096_test_expired_mutation [2022-06-02 13:41:33,046] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:41:38,052] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:41:43,065] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:41:48,076] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:41:53,088] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:41:58,100] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:03,108] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:08,120] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:13,130] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:18,142] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:23,155] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:28,162] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:33,170] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:38,180] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:43,192] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:48,205] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:53,669] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:42:58,680] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:03,692] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:08,705] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:13,712] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:18,724] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:23,736] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:28,748] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:33,759] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:38,772] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:43,784] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:48,796] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:53,804] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:43:58,816] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:03,828] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:08,840] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:13,852] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:18,864] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:23,872] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:28,881] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:33,897] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:38,908] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:44,917] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:49,929] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:44:54,940] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:00,068] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:05,081] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:10,093] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:15,103] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:20,116] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:25,128] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:30,140] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:35,152] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:40,165] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:45,177] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:50,189] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:45:55,201] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:00,213] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:05,225] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:10,238] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:15,250] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:20,263] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:25,271] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:31,282] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:36,294] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:41,302] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:46,315] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:51,326] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:46:56,335] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:01,345] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:06,353] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:11,366] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:16,378] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:21,388] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:26,399] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:31,409] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:36,421] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:41,434] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:46,446] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:51,458] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:47:56,470] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:04,482] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:09,489] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:14,501] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:19,510] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:24,522] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:29,534] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:34,545] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:39,557] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:44,570] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:49,581] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:54,591] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:48:59,603] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:04,611] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:09,621] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:14,629] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:19,636] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:24,648] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:29,663] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:34,674] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:39,685] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:44,695] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:49,704] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:54,716] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:49:59,728] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:04,741] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:09,753] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:14,760] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:19,772] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:24,785] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:32,805] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:37,816] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:42,830] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:47,842] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:52,854] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:50:57,861] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:02,873] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:07,885] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:12,893] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:17,904] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:22,916] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:27,926] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:32,937] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:37,949] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:42,961] - [basetestcase] [139727030187776] - INFO - sleep for 5 secs. Waiting for Function_995760096_test_expired_mutation to undeployed... ... [2022-06-02 13:51:47,973] - [eventing_base] [139727030187776] - INFO - ============== EventingBaseTest tearDown has started ============== [2022-06-02 13:51:47,991] - [basetestcase] [139727030187776] - INFO - list of eventing nodes in cluster: [ip:172.23.122.221 port:8091 ssh_username:root] [2022-06-02 13:51:47,992] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:51:48,099] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:51:48,373] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:51:48,627] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:51:48,635] - [rest_client] [139727030187776] - INFO - /diag/eval status on 172.23.122.221:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-06-02 13:51:48,635] - [remote_util] [139727030187776] - INFO - running command.raw on 172.23.122.221: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2022-06-02 13:51:48,690] - [remote_util] [139727030187776] - INFO - command executed successfully with root [2022-06-02 13:51:48,690] - [remote_util] [139727030187776] - INFO - running command.raw on 172.23.122.221: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2022-06-02 13:51:48,737] - [remote_util] [139727030187776] - INFO - command executed successfully with root [2022-06-02 13:51:48,778] - [basetestcase] [139727030187776] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.208 port:8091 ssh_username:root] [2022-06-02 13:51:48,784] - [rest_client] [139727030187776] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2022-06-02 13:51:48,800] - [eventing_base] [139727030187776] - INFO - number of documents in metadata bucket 0 [2022-06-02 13:51:48,808] - [basetestcase] [139727030187776] - INFO - ------- Cluster statistics ------- [2022-06-02 13:51:48,808] - [basetestcase] [139727030187776] - INFO - 172.23.121.143:8091 => {'services': ['kv'], 'cpu_utilization': 3.309408472845713, 'mem_free': 10168262656, 'mem_total': 12294119424, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:51:48,808] - [basetestcase] [139727030187776] - INFO - 172.23.121.79:8091 => {'services': ['kv'], 'cpu_utilization': 2.51143801224821, 'mem_free': 9853923328, 'mem_total': 12294160384, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:51:48,808] - [basetestcase] [139727030187776] - INFO - 172.23.122.208:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.8989342490606609, 'mem_free': 11210428416, 'mem_total': 12294164480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:51:48,809] - [basetestcase] [139727030187776] - INFO - 172.23.122.221:8091 => {'services': ['eventing'], 'cpu_utilization': 0.9320648872119169, 'mem_free': 10981388288, 'mem_total': 12294160384, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-06-02 13:51:48,809] - [basetestcase] [139727030187776] - INFO - --- End of cluster statistics --- [2022-06-02 13:51:48,810] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:51:48,912] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:51:49,198] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:51:49,462] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:51:49,464] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.79 with username:root, attempt#1 of 5 [2022-06-02 13:51:49,574] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.79 as root [2022-06-02 13:51:49,869] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:51:50,131] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:51:50,133] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:51:50,233] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:51:50,501] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:51:50,753] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:51:50,755] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 [2022-06-02 13:51:50,867] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.122.208 as root [2022-06-02 13:51:51,165] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:51:51,427] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:52:00,559] - [basetestcase] [139727030187776] - INFO - ============== basetestcase cleanup was started for test #2 test_expired_mutation ============== [2022-06-02 13:52:00,574] - [basetestcase] [139727030187776] - WARNING - Alerts were found: [{'msg': 'Hard out-of-memory error: Bucket "dst_bucket" on node 172.23.121.143 is full. No memory currently allocated to this bucket can be easily released.', 'serverTime': '2022-06-02T12:21:53.000Z', 'disableUIPopUp': False}, {'msg': 'Hard out-of-memory error: Bucket "dst_bucket" on node 172.23.121.79 is full. No memory currently allocated to this bucket can be easily released.', 'serverTime': '2022-06-02T12:37:17.000Z', 'disableUIPopUp': False}, {'msg': 'Hard out-of-memory error: Bucket "src_bucket" on node 172.23.121.79 is full. No memory currently allocated to this bucket can be easily released.', 'serverTime': '2022-06-02T13:15:22.000Z', 'disableUIPopUp': False}] [2022-06-02 13:52:00,610] - [bucket_helper] [139727030187776] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.121.143 [2022-06-02 13:52:31,362] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/dst_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{"_":"Bucket deletion not yet complete, but will continue.\\r\\n"}' auth: Administrator:password [2022-06-02 13:52:31,362] - [rest_client] [139727030187776] - WARNING - Bucket deletion timed out waiting for all nodes, retrying... [2022-06-02 13:52:36,395] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/dst_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: status: 404, content: b'Requested resource not found.\r\n' b'Requested resource not found.\r\n' auth: Administrator:password [2022-06-02 13:52:36,396] - [rest_client] [139727030187776] - WARNING - Bucket does not exist..exiting bucket delete [2022-06-02 13:52:36,405] - [bucket_helper] [139727030187776] - ERROR - Unable to get timings for bucket [2022-06-02 13:52:36,405] - [bucket_helper] [139727030187776] - INFO - waiting for bucket deletion to complete.... [2022-06-02 13:52:36,425] - [rest_client] [139727030187776] - INFO - node 172.23.121.143 existing buckets : ['metadata', 'src_bucket'] [2022-06-02 13:52:36,427] - [bucket_helper] [139727030187776] - INFO - deleted bucket : dst_bucket from 172.23.121.143 [2022-06-02 13:53:06,791] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/metadata body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{"_":"Bucket deletion not yet complete, but will continue.\\r\\n"}' auth: Administrator:password [2022-06-02 13:53:06,792] - [rest_client] [139727030187776] - WARNING - Bucket deletion timed out waiting for all nodes, retrying... [2022-06-02 13:53:11,800] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/metadata body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: status: 404, content: b'Requested resource not found.\r\n' b'Requested resource not found.\r\n' auth: Administrator:password [2022-06-02 13:53:11,801] - [rest_client] [139727030187776] - WARNING - Bucket does not exist..exiting bucket delete [2022-06-02 13:53:11,810] - [bucket_helper] [139727030187776] - ERROR - Unable to get timings for bucket [2022-06-02 13:53:11,810] - [bucket_helper] [139727030187776] - INFO - waiting for bucket deletion to complete.... [2022-06-02 13:53:11,821] - [rest_client] [139727030187776] - INFO - node 172.23.121.143 existing buckets : ['src_bucket'] [2022-06-02 13:53:11,821] - [bucket_helper] [139727030187776] - INFO - deleted bucket : metadata from 172.23.121.143 [2022-06-02 13:53:41,944] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/src_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{"_":"Bucket deletion not yet complete, but will continue.\\r\\n"}' auth: Administrator:password [2022-06-02 13:53:41,944] - [rest_client] [139727030187776] - WARNING - Bucket deletion timed out waiting for all nodes, retrying... [2022-06-02 13:53:46,952] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/pools/default/buckets/src_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: status: 404, content: b'Requested resource not found.\r\n' b'Requested resource not found.\r\n' auth: Administrator:password [2022-06-02 13:53:46,953] - [rest_client] [139727030187776] - WARNING - Bucket does not exist..exiting bucket delete [2022-06-02 13:53:46,961] - [bucket_helper] [139727030187776] - ERROR - Unable to get timings for bucket [2022-06-02 13:53:46,961] - [bucket_helper] [139727030187776] - INFO - waiting for bucket deletion to complete.... [2022-06-02 13:53:46,963] - [rest_client] [139727030187776] - INFO - node 172.23.121.143 existing buckets : [] [2022-06-02 13:53:46,963] - [bucket_helper] [139727030187776] - INFO - deleted bucket : src_bucket from 172.23.121.143 [2022-06-02 13:53:46,972] - [bucket_helper] [139727030187776] - INFO - Could not find any buckets for node 172.23.121.79, nothing to delete [2022-06-02 13:53:46,981] - [bucket_helper] [139727030187776] - INFO - Could not find any buckets for node 172.23.122.221, nothing to delete [2022-06-02 13:53:46,991] - [bucket_helper] [139727030187776] - INFO - Could not find any buckets for node 172.23.122.208, nothing to delete [2022-06-02 13:53:46,994] - [basetestcase] [139727030187776] - INFO - Removing user 'clientuser'... [2022-06-02 13:53:47,007] - [rest_client] [139727030187776] - ERROR - DELETE http://172.23.121.143:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-06-02 13:53:47,008] - [basetestcase] [139727030187776] - INFO - b'"User was not found."' [2022-06-02 13:53:47,011] - [rest_client] [139727030187776] - INFO - -->is_ns_server_running? [2022-06-02 13:53:47,022] - [cluster_helper] [139727030187776] - INFO - rebalancing all nodes in order to remove nodes [2022-06-02 13:53:47,024] - [rest_client] [139727030187776] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.121.143,ns_1@172.23.121.79,ns_1@172.23.122.208,ns_1@172.23.122.221', 'ejectedNodes': 'ns_1@172.23.121.79,ns_1@172.23.122.208,ns_1@172.23.122.221', 'user': 'Administrator', 'password': 'password'} [2022-06-02 13:53:47,586] - [rest_client] [139727030187776] - INFO - rebalance operation started [2022-06-02 13:53:47,589] - [rest_client] [139727030187776] - INFO - rebalance percentage : 0.00 % [2022-06-02 13:53:57,602] - [rest_client] [139727030187776] - INFO - rebalance percentage : 62.00 % [2022-06-02 13:54:07,616] - [rest_client] [139727030187776] - INFO - rebalance percentage : 87.00 % [2022-06-02 13:54:20,031] - [rest_client] [139727030187776] - INFO - rebalance percentage : 87.00 % [2022-06-02 13:54:40,050] - [rest_client] [139727030187776] - INFO - rebalance progress took 52.46 seconds [2022-06-02 13:54:40,050] - [rest_client] [139727030187776] - INFO - sleep for 10 seconds after rebalance... [2022-06-02 13:54:50,064] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:54:50,065] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:54:55,072] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:54:55,072] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:55:00,080] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:55:00,081] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:55:05,082] - [rest_client] [139727030187776] - ERROR - node 172.23.121.79:8091 is in a broken state! [2022-06-02 13:55:05,082] - [cluster_helper] [139727030187776] - ERROR - can't create rest connection after rebalance out for ejected nodes, will retry after 10 seconds according to MB-8430: unable to reach the host @ 172.23.121.79 [2022-06-02 13:55:15,095] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:55:15,096] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:55:20,103] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:55:20,104] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:55:25,111] - [rest_client] [139727030187776] - ERROR - GET http://172.23.121.79:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2022-06-02 13:55:25,112] - [rest_client] [139727030187776] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2022-06-02 13:55:30,117] - [rest_client] [139727030187776] - ERROR - node 172.23.121.79:8091 is in a broken state! [2022-06-02 13:55:30,118] - [basetestcase] [139727030187776] - INFO - closing all ssh connections [2022-06-02 13:55:30,118] - [basetestcase] [139727030187776] - INFO - closing all memcached connections [2022-06-02 13:55:30,127] - [eventing_base] [139727030187776] - INFO - ============== EventingBaseTest tearDown has completed ============== [2022-06-02 13:55:30,136] - [remote_util] [139727030187776] - INFO - SSH Connecting to 172.23.121.143 with username:root, attempt#1 of 5 [2022-06-02 13:55:30,137] - [remote_util] [139727292212992] - INFO - SSH Connecting to 172.23.121.79 with username:root, attempt#1 of 5 [2022-06-02 13:55:30,138] - [remote_util] [139727112230656] - INFO - SSH Connecting to 172.23.122.221 with username:root, attempt#1 of 5 [2022-06-02 13:55:30,139] - [remote_util] [139728115017472] - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 [2022-06-02 13:55:30,247] - [remote_util] [139727030187776] - INFO - SSH Connected to 172.23.121.143 as root [2022-06-02 13:55:30,248] - [remote_util] [139727112230656] - INFO - SSH Connected to 172.23.122.221 as root [2022-06-02 13:55:30,252] - [remote_util] [139728115017472] - INFO - SSH Connected to 172.23.122.208 as root [2022-06-02 13:55:30,254] - [remote_util] [139727292212992] - INFO - SSH Connected to 172.23.121.79 as root [2022-06-02 13:55:30,508] - [remote_util] [139727030187776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:55:30,537] - [remote_util] [139727112230656] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:55:30,546] - [remote_util] [139728115017472] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:55:30,549] - [remote_util] [139727292212992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-06-02 13:55:30,797] - [remote_util] [139727112230656] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:55:30,798] - [remote_util] [139727112230656] - INFO - running command.raw on 172.23.122.221: /opt/couchbase/bin/cbcollect_info 172.23.122.221-20220602-1355-diag.zip [2022-06-02 13:55:30,806] - [remote_util] [139728115017472] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:55:30,807] - [remote_util] [139728115017472] - INFO - running command.raw on 172.23.122.208: /opt/couchbase/bin/cbcollect_info 172.23.122.208-20220602-1355-diag.zip [2022-06-02 13:55:30,821] - [remote_util] [139727030187776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:55:30,822] - [remote_util] [139727030187776] - INFO - running command.raw on 172.23.121.143: /opt/couchbase/bin/cbcollect_info 172.23.121.143-20220602-1355-diag.zip [2022-06-02 13:55:30,850] - [remote_util] [139727292212992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-06-02 13:55:30,851] - [remote_util] [139727292212992] - INFO - running command.raw on 172.23.121.79: /opt/couchbase/bin/cbcollect_info 172.23.121.79-20220602-1355-diag.zip [2022-06-02 13:57:25,994] - [remote_util] [139728115017472] - INFO - command executed successfully with root [2022-06-02 13:57:26,195] - [remote_util] [139728115017472] - INFO - running command.raw on 172.23.122.208: rm -f /root/172.23.122.208-20220602-1355-diag.zip [2022-06-02 13:57:26,204] - [remote_util] [139728115017472] - INFO - command executed successfully with root [2022-06-02 13:57:27,097] - [remote_util] [139727112230656] - INFO - command executed successfully with root [2022-06-02 13:57:27,323] - [remote_util] [139727112230656] - INFO - running command.raw on 172.23.122.221: rm -f /root/172.23.122.221-20220602-1355-diag.zip [2022-06-02 13:57:27,331] - [remote_util] [139727112230656] - INFO - command executed successfully with root [2022-06-02 13:57:29,210] - [remote_util] [139727292212992] - INFO - command executed successfully with root [2022-06-02 13:57:29,540] - [remote_util] [139727292212992] - INFO - running command.raw on 172.23.121.79: rm -f /root/172.23.121.79-20220602-1355-diag.zip [2022-06-02 13:57:29,551] - [remote_util] [139727292212992] - INFO - command executed successfully with root [2022-06-02 13:58:01,277] - [remote_util] [139727030187776] - INFO - command executed successfully with root [2022-06-02 13:58:01,874] - [remote_util] [139727030187776] - INFO - running command.raw on 172.23.121.143: rm -f /root/172.23.121.143-20220602-1355-diag.zip [2022-06-02 13:58:01,928] - [remote_util] [139727030187776] - INFO - command executed successfully with root