[2020-05-27 05:00:24,901] - [remote_util] [140290839557888] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:24,997] - [remote_util] [140290839557888] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:25,193] - [remote_util] [140290839557888] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:25,486] - [remote_util] [140290839557888] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:25,493] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:25,506] - [rest_client] [140290839557888] - INFO - Node version in cluster 7.0.0-2170-enterprise [2020-05-27 05:00:25,514] - [rest_client] [140290839557888] - INFO - Node versions in cluster ['7.0.0-2170-enterprise'] [2020-05-27 05:00:25,514] - [basetestcase] [140290839557888] - INFO - ============== basetestcase setup was started for test #17 test_sanity============== [2020-05-27 05:00:25,515] - [basetestcase] [140290839557888] - INFO - any cluster operation in setup will be skipped [2020-05-27 05:00:25,515] - [basetestcase] [140290839557888] - INFO - ============== basetestcase setup was finished for test #17 test_sanity ============== [2020-05-27 05:00:25,516] - [remote_util] [140290839557888] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:25,609] - [remote_util] [140290839557888] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:25,842] - [remote_util] [140290839557888] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:26,146] - [remote_util] [140290839557888] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:26,152] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:26,163] - [rest_client] [140290839557888] - INFO - Node version in cluster 7.0.0-2170-enterprise [2020-05-27 05:00:26,169] - [rest_client] [140290839557888] - INFO - Node versions in cluster ['7.0.0-2170-enterprise'] [2020-05-27 05:00:26,170] - [basetestcase] [140290839557888] - INFO - ============== basetestcase setup was started for test #17 test_sanity============== [2020-05-27 05:00:26,184] - [bucket_helper] [140290839557888] - INFO - Could not find any buckets for node 172.23.107.190, nothing to delete [2020-05-27 05:00:26,188] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:26,202] - [cluster_helper] [140290839557888] - INFO - waiting for ns_server @ 172.23.107.190:8091 [2020-05-27 05:00:26,202] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:26,206] - [cluster_helper] [140290839557888] - INFO - ns_server @ 172.23.107.190:8091 is running [2020-05-27 05:00:26,207] - [basetestcase] [140290839557888] - INFO - initializing cluster [2020-05-27 05:00:26,908] - [task] [140290518140672] - INFO - server: ip:172.23.107.190 port:8091 ssh_username:root, nodes/self [2020-05-27 05:00:26,912] - [task] [140290518140672] - INFO - {'uptime': '584', 'memoryTotal': 7934996480, 'memoryFree': 6541266944, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.190:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2170-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3287, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.190', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7567, 'curr_items': 0} [2020-05-27 05:00:26,917] - [task] [140290518140672] - INFO - quota for index service will be 256 MB [2020-05-27 05:00:26,917] - [task] [140290518140672] - INFO - set index quota to node 172.23.107.190 [2020-05-27 05:00:26,917] - [rest_client] [140290518140672] - INFO - pools/default params : indexMemoryQuota=256 [2020-05-27 05:00:26,922] - [task] [140290518140672] - INFO - quota for fts service will be 512 MB [2020-05-27 05:00:26,922] - [task] [140290518140672] - INFO - set both index and fts quota at node 172.23.107.190 [2020-05-27 05:00:26,922] - [rest_client] [140290518140672] - INFO - pools/default params : ftsMemoryQuota=512 [2020-05-27 05:00:26,929] - [rest_client] [140290518140672] - INFO - pools/default params : memoryQuota=3287 [2020-05-27 05:00:26,938] - [rest_client] [140290518140672] - INFO - --> init_node_services(Administrator,password,172.23.107.190,8091,['kv', 'index', 'n1ql', 'fts']) [2020-05-27 05:00:26,938] - [rest_client] [140290518140672] - INFO - /node/controller/setupServices params on 172.23.107.190: 8091:hostname=172.23.107.190&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2020-05-27 05:00:26,941] - [rest_client] [140290518140672] - ERROR - POST http://172.23.107.190:8091//node/controller/setupServices body: hostname=172.23.107.190&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2020-05-27 05:00:26,941] - [rest_client] [140290518140672] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2020-05-27 05:00:26,941] - [rest_client] [140290518140672] - INFO - --> in init_cluster...Administrator,password,8091 [2020-05-27 05:00:26,942] - [rest_client] [140290518140672] - INFO - settings/web params on 172.23.107.190:8091:port=8091&username=Administrator&password=password [2020-05-27 05:00:26,997] - [rest_client] [140290518140672] - INFO - --> status:True [2020-05-27 05:00:26,998] - [remote_util] [140290518140672] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:27,093] - [remote_util] [140290518140672] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:27,292] - [remote_util] [140290518140672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:27,592] - [remote_util] [140290518140672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:27,604] - [remote_util] [140290518140672] - INFO - running command.raw on 172.23.107.190: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-05-27 05:00:27,801] - [remote_util] [140290518140672] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 13 266 --:--:-- --:--:-- --:--:-- 267'] ... [2020-05-27 05:00:27,801] - [remote_util] [140290518140672] - INFO - ['ok'] [2020-05-27 05:00:27,804] - [rest_client] [140290518140672] - INFO - /diag/eval status on 172.23.107.190:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-05-27 05:00:27,806] - [rest_client] [140290518140672] - INFO - /diag/eval status on 172.23.107.190:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-05-27 05:00:27,806] - [rest_client] [140290518140672] - INFO - settings/indexes params : storageMode=plasma [2020-05-27 05:00:27,833] - [basetestcase] [140290839557888] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.107.190 **** [2020-05-27 05:00:27,940] - [basetestcase] [140290839557888] - INFO - sleep for 5 secs. ... [2020-05-27 05:00:32,945] - [basetestcase] [140290839557888] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-05-27 05:00:32,961] - [basetestcase] [140290839557888] - INFO - done initializing cluster [2020-05-27 05:00:32,963] - [remote_util] [140290839557888] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:33,056] - [remote_util] [140290839557888] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:33,254] - [remote_util] [140290839557888] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:33,551] - [remote_util] [140290839557888] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:33,562] - [remote_util] [140290839557888] - INFO - running command.raw on 172.23.107.190: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-05-27 05:00:33,759] - [remote_util] [140290839557888] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 12 265 --:--:-- --:--:-- --:--:-- 266'] ... [2020-05-27 05:00:33,760] - [remote_util] [140290839557888] - INFO - ['ok'] [2020-05-27 05:00:33,760] - [basetestcase] [140290839557888] - INFO - Enabled diag/eval for non-local hosts from 172.23.107.190 [2020-05-27 05:00:33,853] - [rest_client] [140290518140672] - INFO - http://172.23.107.190:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=3287&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive [2020-05-27 05:00:33,858] - [rest_client] [140290518140672] - INFO - 0.01 seconds to create bucket default [2020-05-27 05:00:33,859] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:34,161] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,287] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,338] - [task] [140290518140672] - WARNING - vbucket map not ready after try 0 [2020-05-27 05:00:34,338] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:34,449] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,535] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,587] - [task] [140290518140672] - WARNING - vbucket map not ready after try 1 [2020-05-27 05:00:34,587] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:34,709] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,832] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:34,884] - [task] [140290518140672] - WARNING - vbucket map not ready after try 2 [2020-05-27 05:00:34,885] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:34,974] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,098] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,153] - [task] [140290518140672] - WARNING - vbucket map not ready after try 3 [2020-05-27 05:00:35,153] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:35,267] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,398] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,452] - [task] [140290518140672] - WARNING - vbucket map not ready after try 4 [2020-05-27 05:00:35,452] - [bucket_helper] [140290518140672] - INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops [2020-05-27 05:00:35,539] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,661] - [data_helper] [140290518140672] - INFO - creating direct client 172.23.107.190:11210 default [2020-05-27 05:00:35,719] - [task] [140290518140672] - WARNING - vbucket map not ready after try 5 [2020-05-27 05:00:35,721] - [basetestcase] [140290839557888] - INFO - ============== basetestcase setup was finished for test #17 test_sanity ============== [2020-05-27 05:00:35,735] - [basetestcase] [140290839557888] - INFO - ------- Cluster statistics ------- [2020-05-27 05:00:35,735] - [basetestcase] [140290839557888] - INFO - 172.23.107.190:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 20.72538860103627, 'mem_free': 6615560192, 'mem_total': 7934996480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2020-05-27 05:00:35,736] - [basetestcase] [140290839557888] - INFO - --- End of cluster statistics --- [2020-05-27 05:00:35,736] - [tuq] [140290839557888] - INFO - ============== QueryTests setup has started ============== [2020-05-27 05:00:35,737] - [remote_util] [140290839557888] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:35,828] - [remote_util] [140290839557888] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:36,030] - [remote_util] [140290839557888] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:36,326] - [remote_util] [140290839557888] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:36,370] - [remote_util] [140290839557888] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:36,464] - [remote_util] [140290839557888] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:36,665] - [remote_util] [140290839557888] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:36,965] - [remote_util] [140290839557888] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:36,967] - [tuq] [140290839557888] - INFO - Generating default:default data... [2020-05-27 05:00:36,967] - [tuq] [140290839557888] - INFO - -->start:generate_docs_employee for default... [2020-05-27 05:00:37,028] - [tuq] [140290839557888] - INFO - -->end:generate_docs_employee for default... [2020-05-27 05:00:37,029] - [tuq] [140290839557888] - INFO - Completed Generating default:default data... [2020-05-27 05:00:37,029] - [tuq] [140290839557888] - INFO - --->gens_load is done... [2020-05-27 05:00:37,029] - [tuq] [140290839557888] - INFO - -->starting rest connection... [2020-05-27 05:00:47,029] - [tuq] [140290839557888] - INFO - --> docs_per_day>0..generating TuqGenerators... [2020-05-27 05:00:57,206] - [tuq] [140290839557888] - INFO - --> End: docs_per_day>0..generating TuqGenerators... [2020-05-27 05:00:57,207] - [tuq] [140290839557888] - INFO - ============== QueryTests setup has completed ============== [2020-05-27 05:00:57,207] - [tuq_dml] [140290839557888] - INFO - ============== DMLQueryTests setup has started ============== [2020-05-27 05:00:57,207] - [tuq] [140290839557888] - INFO - xxxxxx - - - All buckets - - - xxxxxx [2020-05-27 05:00:57,207] - [tuq] [140290839557888] - INFO - ['default'] [2020-05-27 05:00:57,207] - [tuq_dml] [140290839557888] - INFO - ---------------------------------------------------------------------------------------------------- [2020-05-27 05:00:57,207] - [tuq_dml] [140290839557888] - INFO - ============== DMLQueryTests setup has started ============== [2020-05-27 05:00:57,213] - [tuq] [140290839557888] - INFO - RUN QUERY SELECT * FROM system:indexes [2020-05-27 05:00:57,214] - [rest_client] [140290839557888] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,286] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 70.531537ms [2020-05-27 05:00:57,292] - [tuq] [140290839557888] - INFO - RUN QUERY SELECT * FROM system:keyspaces [2020-05-27 05:00:57,292] - [rest_client] [140290839557888] - INFO - query params : statement=SELECT+%2A+FROM+system%3Akeyspaces&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,306] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 12.584714ms [2020-05-27 05:00:57,306] - [tuq] [140290839557888] - INFO - ============== System Config: ============== [2020-05-27 05:00:57,311] - [tuq] [140290839557888] - INFO - RUN QUERY SELECT COUNT(*) FROM `default` [2020-05-27 05:00:57,311] - [rest_client] [140290839557888] - INFO - query params : statement=SELECT+COUNT%28%2A%29+FROM+%60default%60&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,323] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 10.18834ms [2020-05-27 05:00:57,323] - [tuq] [140290839557888] - INFO - Bucket: default [2020-05-27 05:00:57,323] - [tuq] [140290839557888] - INFO - Indexes: [] [2020-05-27 05:00:57,323] - [tuq] [140290839557888] - INFO - Docs: 0 [2020-05-27 05:00:57,323] - [tuq] [140290839557888] - INFO - ============================================= [2020-05-27 05:00:57,329] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-0", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 1, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-0'}) [2020-05-27 05:00:57,329] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-0%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+1%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-0%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,339] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 7.645482ms [2020-05-27 05:00:57,344] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-1", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 2, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-1'}) [2020-05-27 05:00:57,345] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-1%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+2%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-1%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,351] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 5.016028ms [2020-05-27 05:00:57,356] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-2", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 3, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-2'}) [2020-05-27 05:00:57,357] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-2%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+3%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-2%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,365] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 5.838352ms [2020-05-27 05:00:57,370] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-3", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 4, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-3'}) [2020-05-27 05:00:57,371] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-3%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+4%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-3%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,377] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 4.606905ms [2020-05-27 05:00:57,382] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-4", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 5, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-4'}) [2020-05-27 05:00:57,383] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-4%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+5%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-4%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,390] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 5.378337ms [2020-05-27 05:00:57,395] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-5", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 6, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-5'}) [2020-05-27 05:00:57,396] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-5%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+6%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-5%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,404] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 6.055702ms [2020-05-27 05:00:57,409] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-6", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 7, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-6'}) [2020-05-27 05:00:57,410] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-6%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+7%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-6%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,417] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 5.070039ms [2020-05-27 05:00:57,422] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-7", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 8, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-7'}) [2020-05-27 05:00:57,422] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-7%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+8%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-7%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,431] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 7.339906ms [2020-05-27 05:00:57,437] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-8", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 9, 'task2': 1}, '_id': 'query-testemployee13436.424411240123-8'}) [2020-05-27 05:00:57,437] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-8%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+9%2C+%27task2%27%3A+1%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-8%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,444] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 5.138777ms [2020-05-27 05:00:57,449] - [tuq] [140290839557888] - INFO - RUN QUERY INSERT into default (key , value) VALUES ("update_wheree9f5query-testemployee13436.424411240123-9", {'name': 'employee-1', 'join_yr': 2010, 'join_mo': 1, 'join_day': 1, 'email': '1-mail@couchbase.com', 'job_title': 'Engineer', 'test_rate': 1.1, 'skills': ['skill2010', 'skill2011'], 'VMs': [{'RAM': 1, 'os': 'ubuntu', 'name': 'vm_1', 'memory': 1}, {'RAM': 1, 'os': 'windows', 'name': 'vm_2', 'memory': 1}], 'tasks_points': {'task1': 1, 'task2': 2}, '_id': 'query-testemployee13436.424411240123-9'}) [2020-05-27 05:00:57,449] - [rest_client] [140290839557888] - INFO - query params : statement=INSERT+into+default+%28key+%2C+value%29+VALUES+%28%22update_wheree9f5query-testemployee13436.424411240123-9%22%2C+%7B%27name%27%3A+%27employee-1%27%2C+%27join_yr%27%3A+2010%2C+%27join_mo%27%3A+1%2C+%27join_day%27%3A+1%2C+%27email%27%3A+%271-mail%40couchbase.com%27%2C+%27job_title%27%3A+%27Engineer%27%2C+%27test_rate%27%3A+1.1%2C+%27skills%27%3A+%5B%27skill2010%27%2C+%27skill2011%27%5D%2C+%27VMs%27%3A+%5B%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27ubuntu%27%2C+%27name%27%3A+%27vm_1%27%2C+%27memory%27%3A+1%7D%2C+%7B%27RAM%27%3A+1%2C+%27os%27%3A+%27windows%27%2C+%27name%27%3A+%27vm_2%27%2C+%27memory%27%3A+1%7D%5D%2C+%27tasks_points%27%3A+%7B%27task1%27%3A+1%2C+%27task2%27%3A+2%7D%2C+%27_id%27%3A+%27query-testemployee13436.424411240123-9%27%7D%29&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,455] - [tuq] [140290839557888] - INFO - TOTAL ELAPSED TIME: 4.609168ms [2020-05-27 05:00:57,460] - [tuq] [140290839557888] - INFO - RUN QUERY update default set name="new_name" where join_day=1 returning name [2020-05-27 05:00:57,461] - [rest_client] [140290839557888] - INFO - query params : statement=update+default+set+name%3D%22new_name%22+where+join_day%3D1+returning+name&scan_consistency=REQUEST_PLUS [2020-05-27 05:00:57,471] - [rest_client] [140290839557888] - ERROR - POST http://172.23.107.190:8093/query?statement=update+default+set+name%3D%22new_name%22+where+join_day%3D1+returning+name&scan_consistency=REQUEST_PLUS body: headers: {'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=='} error: 404 reason: unknown b'{\n"requestID": "3a20242e-9c2a-426b-886b-36cd75234c9d",\n"errors": [{"code":4000,"msg":"No index available on keyspace default that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online."}],\n"status": "fatal",\n"metrics": {"elapsedTime": "9.121903ms","executionTime": "9.058117ms","resultCount": 0,"resultSize": 0,"errorCount": 1}\n}\n' auth: Administrator:password [2020-05-27 05:00:57,473] - [tuq_dml] [140290839557888] - INFO - ============== DMLQueryTests tearDown has started ============== [2020-05-27 05:00:57,473] - [tuq_dml] [140290839557888] - INFO - ============== DMLQueryTests tearDown has started ============== [2020-05-27 05:00:57,473] - [tuq] [140290839557888] - INFO - ============== QueryTests tearDown has started ============== [2020-05-27 05:00:57,474] - [tuq] [140290839557888] - INFO - ============== QueryTests tearDown has completed ============== [2020-05-27 05:00:57,485] - [basetestcase] [140290839557888] - INFO - ------- Cluster statistics ------- [2020-05-27 05:00:57,485] - [basetestcase] [140290839557888] - INFO - 172.23.107.190:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.526448362720403, 'mem_free': 6675787776, 'mem_total': 7934996480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2020-05-27 05:00:57,485] - [basetestcase] [140290839557888] - INFO - --- End of cluster statistics --- [2020-05-27 05:00:57,485] - [basetestcase] [140290839557888] - INFO - ============== basetestcase cleanup was started for test #17 test_sanity ============== [2020-05-27 05:00:57,512] - [bucket_helper] [140290839557888] - INFO - deleting existing buckets ['default'] on 172.23.107.190 [2020-05-27 05:00:58,150] - [bucket_helper] [140290839557888] - INFO - waiting for bucket deletion to complete.... [2020-05-27 05:00:58,153] - [rest_client] [140290839557888] - INFO - node 172.23.107.190 existing buckets : [] [2020-05-27 05:00:58,153] - [bucket_helper] [140290839557888] - INFO - deleted bucket : default from 172.23.107.190 [2020-05-27 05:00:58,157] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:58,171] - [cluster_helper] [140290839557888] - INFO - waiting for ns_server @ 172.23.107.190:8091 [2020-05-27 05:00:58,171] - [rest_client] [140290839557888] - INFO - -->is_ns_server_running? [2020-05-27 05:00:58,175] - [cluster_helper] [140290839557888] - INFO - ns_server @ 172.23.107.190:8091 is running [2020-05-27 05:00:58,175] - [basetestcase] [140290839557888] - INFO - ============== basetestcase cleanup was finished for test #17 test_sanity ============== [2020-05-27 05:00:58,175] - [basetestcase] [140290839557888] - INFO - closing all ssh connections [2020-05-27 05:00:58,175] - [basetestcase] [140290839557888] - INFO - closing all memcached connections [2020-05-27 05:00:58,182] - [remote_util] [140291237517120] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:00:58,275] - [remote_util] [140291237517120] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:00:58,489] - [remote_util] [140291237517120] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:00:58,795] - [remote_util] [140291237517120] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:00:58,796] - [remote_util] [140291237517120] - INFO - running command.raw on 172.23.107.190: /opt/couchbase/bin/cbcollect_info 172.23.107.190-20200527-0500-diag.zip [2020-05-27 05:03:04,532] - [remote_util] [140291237517120] - INFO - command executed successfully with root [2020-05-27 05:03:04,963] - [remote_util] [140291237517120] - INFO - running command.raw on 172.23.107.190: rm -f /root/172.23.107.190-20200527-0500-diag.zip [2020-05-27 05:03:04,979] - [remote_util] [140291237517120] - INFO - command executed successfully with root [2020-05-27 05:03:04,981] - [remote_util] [140291237517120] - INFO - SSH Connecting to 172.23.107.190 with username:root, attempt#1 of 5 [2020-05-27 05:03:05,075] - [remote_util] [140291237517120] - INFO - SSH Connected to 172.23.107.190 as root [2020-05-27 05:03:05,280] - [remote_util] [140291237517120] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-05-27 05:03:05,576] - [remote_util] [140291237517120] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-05-27 05:03:05,577] - [remote_util] [140291237517120] - INFO - running command.raw on 172.23.107.190: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.190-20200527-0503-couch-dbinfo.txt [2020-05-27 05:03:05,641] - [remote_util] [140291237517120] - INFO - command executed successfully with root [2020-05-27 05:03:05,642] - [remote_util] [140291237517120] - INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file [2020-05-27 05:03:05,731] - [remote_util] [140291237517120] - INFO - File 172.23.107.190-20200527-0503-couch-dbinfo.txt will be deleted [2020-05-27 05:03:05,732] - [remote_util] [140291237517120] - INFO - running command.raw on 172.23.107.190: rm -rf /root/*172.23.107.190-20200527-0503-couch-dbinfo.txt* [2020-05-27 05:03:05,743] - [remote_util] [140291237517120] - INFO - command executed successfully with root [2020-05-27 05:03:05,743] - [remote_util] [140291237517120] - INFO - 172.23.107.190:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ...