/Users/lfc/venv3_with_cb3/bin/python "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 50406 --file /Users/lfc/testrunner/testrunner.py -i b/resources/tmp.ini -p gsi_type=plasma,java_sdk_client=True -t gsi.collections_concurrent_indexes.ConcurrentIndexes.test_drop_indexes_schedule_for_background_creation,default_bucket=false,force_clean=true,defer_build=False,num_of_indexes=10 Connected to pydev debugger (build 203.7717.65) INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.collections_concurrent_indexes.ConcurrentIndexes.test_drop_indexes_schedule_for_background_creation,default_bucket=false,force_clean=true,defer_build=False,num_of_indexes=10 INFO:__main__:Logs folder: /Users/lfc/testrunner/logs/testrunner-21-Mar-24_18-11-11/test_1 {'cluster_name': 'tmp', 'gsi_type': 'plasma', 'ini': 'b/resources/tmp.ini', 'java_sdk_client': 'True', 'num_nodes': 1} Logs will be stored at /Users/lfc/testrunner/logs/testrunner-21-Mar-24_18-11-11/test_1 ./testrunner -i b/resources/tmp.ini -p gsi_type=plasma,java_sdk_client=True -t gsi.collections_concurrent_indexes.ConcurrentIndexes.test_drop_indexes_schedule_for_background_creation,default_bucket=false,force_clean=true,defer_build=False,num_of_indexes=10 Test Input params: {'default_bucket': 'false', 'force_clean': 'true', 'defer_build': 'False', 'num_of_indexes': '10', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'gsi_type': 'plasma', 'java_sdk_client': 'True', 'num_nodes': 1, 'case_number': 1, 'logs_folder': '/Users/lfc/testrunner/logs/testrunner-21-Mar-24_18-11-11/test_1'} Run before suite setup for gsi.collections_concurrent_indexes.ConcurrentIndexes.test_drop_indexes_schedule_for_background_creation -->before_suite_name:gsi.collections_concurrent_indexes.ConcurrentIndexes.suite_setUp,suite: ]> 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 suite_setUp (gsi.collections_concurrent_indexes.ConcurrentIndexes) ... 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4732-enterprise 2021-03-24 18:11:12 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-03-24 18:11:12 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-24 18:11:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has started ============== 2021-03-24 18:11:18 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:18 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 2021-03-24 18:11:18 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-24 18:11:20 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has completed ============== 2021-03-24 18:11:20 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.101 port:8091 ssh_username:root, nodes/self 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '13', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 1841, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '10.112.190.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4732-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-03-24 18:11:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.190.101 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=977 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.190.101,8091,['kv', 'n1ql', 'index']) 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.190.101: 8091:hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.101:8091:port=8091&username=Administrator&password=password 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-24 18:11:22 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-24 18:11:22 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.190.101 **** 2021-03-24 18:11:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://10.112.190.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-24 18:11:22 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-24 18:11:22 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-24 18:11:27 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-24 18:11:27 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-03-24 18:11:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:11:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.190.101 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0, 'mem_free': 0, 'mem_total': 0, 'swap_mem_used': 0, 'swap_mem_total': 0} 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.190.101 cluster is healthy 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.190.101 is healthy 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-24 18:11:30 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-24 18:11:35 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-24 18:11:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:11:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:11:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:11:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:11:43 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2021-03-24 18:11:43 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:11:43 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:10.112.190.101 port:8091 ssh_username:root 2021-03-24 18:11:43 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-24 18:12:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:14 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-03-24 18:12:14 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-03-24 18:12:14 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.setUp] ============== ConcurrentIndexes setup has started ============== 2021-03-24 18:12:15 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.190.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=100&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-24 18:12:15 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket 2021-03-24 18:12:15 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:16 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:16 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:12:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:12:18 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.setUp] ============== ConcurrentIndexes setup has completed ============== 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has started ============== 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-24 18:12:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.57201646090535, 'mem_free': 1489842176, 'mem_total': 1930829824, 'swap_mem_used': 4042752, 'swap_mem_total': 1107292160} 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 10.112.190.101 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.190.101 existing buckets : [] 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 10.112.190.101 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-03-24 18:12:21 | INFO | MainProcess | MainThread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has completed ============== ok ---------------------------------------------------------------------- Ran 1 test in 69.849s OK test_drop_indexes_schedule_for_background_creation (gsi.collections_concurrent_indexes.ConcurrentIndexes) ... -->result: 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4732-enterprise 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-4732-enterprise'] 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_drop_indexes_schedule_for_background_creation============== 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-24 18:12:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has started ============== 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.57201646090535, 'mem_free': 1489842176, 'mem_total': 1930829824, 'swap_mem_used': 4042752, 'swap_mem_total': 1107292160} 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_drop_indexes_schedule_for_background_creation ============== 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_drop_indexes_schedule_for_background_creation ============== 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [collections_concurrent_indexes.tearDown] ============== ConcurrentIndexes tearDown has completed ============== 2021-03-24 18:12:29 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.101 port:8091 ssh_username:root, nodes/self 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '79', 'memoryTotal': 1930829824, 'memoryFree': 1395621888, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.190.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4732-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 977, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.190.101 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.190.101,8091,['kv', 'n1ql', 'index']) 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.190.101: 8091:hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2021-03-24 18:12:30 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.190.101:8091//node/controller/setupServices body: hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.101:8091:port=8091&username=Administrator&password=password 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-24 18:12:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:31 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-24 18:12:31 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-24 18:12:31 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-24 18:12:31 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-24 18:12:31 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.190.101 **** 2021-03-24 18:12:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.190.101 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_drop_indexes_schedule_for_background_creation ============== 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 11.67023554603854, 'mem_free': 1421500416, 'mem_total': 1930829824, 'swap_mem_used': 3969024, 'swap_mem_total': 1107292160} 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.190.101 cluster is healthy 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.190.101 is healthy 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-24 18:12:39 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-24 18:12:44 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-24 18:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:12:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:12:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:12:53 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-03-24 18:12:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-24 18:12:53 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.190.101 port:8091 ssh_username:root 2021-03-24 18:12:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-03-24 18:13:23 | INFO | MainProcess | test_thread | [collections_concurrent_indexes.setUp] ============== ConcurrentIndexes setup has started ============== 2021-03-24 18:13:23 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.190.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=100&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-24 18:13:23 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket 2021-03-24 18:13:23 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:24 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-03-24 18:13:24 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.190.101 to accept set ops 2021-03-24 18:13:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.190.101:11210 test_bucket 2021-03-24 18:13:26 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [collections_concurrent_indexes.setUp] ============== ConcurrentIndexes setup has completed ============== 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.190.101:8091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created test_bucket->test_scope_1 {'uid': '1'} 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.190.101:8091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2021-03-24 18:13:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. Allowing time after collection creation ... 2021-03-24 18:13:36 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] Number of batches : 10 2021-03-24 18:13:36 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-03-24 18:13:46 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] done 2021-03-24 18:13:47 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:14:22 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 10001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:14:57 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 20001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:15:32 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 30001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:16:07 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 40001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:16:42 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 50001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:17:16 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 60001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:17:50 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 70001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:18:24 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 80001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:18:59 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.190.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 90001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o False -sd False 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_0 ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_2 ON default:test_bucket.test_scope_1.test_collection_1(country) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1 ON default:test_bucket.test_scope_1.test_collection_1(city) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_3 ON default:test_bucket.test_scope_1.test_collection_1(title) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_4 ON default:test_bucket.test_scope_1.test_collection_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_5 ON default:test_bucket.test_scope_1.test_collection_1(lastName) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_0+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_7 ON default:test_bucket.test_scope_1.test_collection_1(suffix) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_6 ON default:test_bucket.test_scope_1.test_collection_1(streetAddress) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_8 ON default:test_bucket.test_scope_1.test_collection_1(filler1) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_9 ON default:test_bucket.test_scope_1.test_collection_1(phone) USING GSI WITH {'defer_build': False} 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_3+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28suffix%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28streetAddress%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28lastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28phone%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28filler1%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-24 18:19:15 | ERROR | MainProcess | ThreadPoolExecutor-0_1 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "db03a53d-476f-4e64-ae15-488b044f7f56",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_1, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "267.377927ms","executionTime": "267.349616ms","resultCount": 0,"resultSize": 0,"serviceLoad": 250,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:15 | ERROR | MainProcess | ThreadPoolExecutor-0_4 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "6f27a8a7-68fd-4a49-ad9e-d98267457125",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_4, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "305.488202ms","executionTime": "305.446624ms","resultCount": 0,"resultSize": 0,"serviceLoad": 225,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:15 | ERROR | MainProcess | ThreadPoolExecutor-0_2 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "6e43f1de-cbd2-4b6d-81a0-34076776844c",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_2, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "313.561037ms","executionTime": "313.560843ms","resultCount": 0,"resultSize": 0,"serviceLoad": 200,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:16 | ERROR | MainProcess | ThreadPoolExecutor-0_3 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_3+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "35efab9d-bfc1-4766-a5dc-509ee95a153d",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Encountered transient error. Index creation will be retried in background. Error: Index idx_3 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket:test_scope_1:test_collection_1.\\n"}],\n"status": "errors",\n"metrics": {"elapsedTime": "717.075279ms","executionTime": "450.710151ms","resultCount": 0,"resultSize": 0,"serviceLoad": 175,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:16 | ERROR | MainProcess | ThreadPoolExecutor-0_6 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28streetAddress%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "e63b7159-922c-4cfb-a70e-a4e807fc0ad5",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_6, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "757.990512ms","executionTime": "453.673154ms","resultCount": 0,"resultSize": 0,"serviceLoad": 150,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:16 | ERROR | MainProcess | ThreadPoolExecutor-0_7 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28suffix%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "815be554-bee6-41b8-8ed0-e9f10923fb32",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_7, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "758.306961ms","executionTime": "454.581161ms","resultCount": 0,"resultSize": 0,"serviceLoad": 125,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:16 | ERROR | MainProcess | ThreadPoolExecutor-0_5 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28lastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "b0ec6349-8775-4a20-a179-a4159089e5e0",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Encountered transient error. Index creation will be retried in background. Error: Index idx_5 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket:test_scope_1:test_collection_1.\\n"}],\n"status": "errors",\n"metrics": {"elapsedTime": "867.070172ms","executionTime": "150.066196ms","resultCount": 0,"resultSize": 0,"serviceLoad": 100,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:16 | ERROR | MainProcess | ThreadPoolExecutor-0_9 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28phone%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "7cfc6aea-c7c4-4693-91f6-7d67977addad",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Encountered transient error. Index creation will be retried in background. Error: Index idx_9 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket:test_scope_1:test_collection_1.\\n"}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.004689816s","executionTime": "250.346043ms","resultCount": 0,"resultSize": 0,"serviceLoad": 75,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:17 | ERROR | MainProcess | ThreadPoolExecutor-0_8 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28filler1%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "5fa073f2-9ddf-4435-960d-c0701c84d199",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_8, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "2.11820246s","executionTime": "1.362872549s","resultCount": 0,"resultSize": 0,"serviceLoad": 50,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-24 18:19:25 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.650329116s 2021-03-24 18:19:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_1 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:34 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_1+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 608.346603ms 2021-03-24 18:19:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_2 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:40 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_2+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:41 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 935.328267ms 2021-03-24 18:19:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_4 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_4+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 803.544168ms 2021-03-24 18:19:50 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_6 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:50 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_6+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.667027292s 2021-03-24 18:19:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_7 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_7+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 411.474928ms 2021-03-24 18:19:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY drop index idx_8 on default:test_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:56 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=drop+index+idx_8+on+default%3Atest_bucket.test_scope_1.test_collection_1 2021-03-24 18:19:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 61.99067ms 2021-03-24 18:19:59 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 120 secs. ... 2021-03-24 18:21:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select * from system:indexes 2021-03-24 18:21:59 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+%2A+from+system%3Aindexes 2021-03-24 18:21:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.934929ms 2021-03-24 18:21:59 | INFO | MainProcess | test_thread | [collections_concurrent_indexes.test_drop_indexes_schedule_for_background_creation] [{'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': '1b0c1e9b7be08363', 'index_key': ['`title`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_3', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': '9397a7efc919a145', 'index_key': ['`phone`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_9', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': 'f15056f2643fb40d', 'index_key': ['`suffix`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_7', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': 'ed30d6c637db2165', 'index_key': ['`age`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_0', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': '990e69ef2b6dc353', 'index_key': ['`lastName`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_5', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'bucket_id': 'test_bucket', 'datastore_id': 'http://127.0.0.1:8091', 'id': '185eb55737a1aadf', 'index_key': ['`streetAddress`'], 'keyspace_id': 'test_collection_1', 'name': 'idx_6', 'namespace_id': 'default', 'scope_id': 'test_scope_1', 'state': 'online', 'using': 'gsi'}}]