[2019-11-28 11:16:54,102] - [eventing_base] [140525218207488] - INFO - Starting Test: test_n1ql_timeout [2019-11-28 11:16:54,105] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:16:54,272] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:16:55,223] - [rest_client] [140525218207488] - INFO - Node version in cluster 6.5.0-4897-enterprise [2019-11-28 11:16:55,276] - [rest_client] [140525218207488] - INFO - Node versions in cluster [u'6.5.0-4897-enterprise', u'6.5.0-4897-enterprise', u'6.5.0-4897-enterprise', u'6.5.0-4897-enterprise'] [2019-11-28 11:16:55,277] - [basetestcase] [140525218207488] - INFO - ============== basetestcase setup was started for test #17 test_n1ql_timeout============== [2019-11-28 11:16:55,504] - [bucket_helper] [140525218207488] - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket', u'travel-sample'] on 172.23.104.100 [2019-11-28 11:16:55,507] - [bucket_helper] [140525218207488] - INFO - remove bucket dst_bucket ... [2019-11-28 11:16:56,717] - [bucket_helper] [140525218207488] - INFO - deleted bucket : dst_bucket from 172.23.104.100 [2019-11-28 11:16:56,717] - [bucket_helper] [140525218207488] - INFO - waiting for bucket deletion to complete.... [2019-11-28 11:16:56,857] - [rest_client] [140525218207488] - INFO - node 172.23.104.100 existing buckets : [u'metadata', u'src_bucket', u'travel-sample'] [2019-11-28 11:16:56,862] - [bucket_helper] [140525218207488] - INFO - remove bucket metadata ... [2019-11-28 11:16:58,638] - [bucket_helper] [140525218207488] - INFO - deleted bucket : metadata from 172.23.104.100 [2019-11-28 11:16:58,639] - [bucket_helper] [140525218207488] - INFO - waiting for bucket deletion to complete.... [2019-11-28 11:16:58,702] - [rest_client] [140525218207488] - INFO - node 172.23.104.100 existing buckets : [u'src_bucket', u'travel-sample'] [2019-11-28 11:16:58,707] - [bucket_helper] [140525218207488] - INFO - remove bucket src_bucket ... [2019-11-28 11:17:00,728] - [bucket_helper] [140525218207488] - INFO - deleted bucket : src_bucket from 172.23.104.100 [2019-11-28 11:17:00,729] - [bucket_helper] [140525218207488] - INFO - waiting for bucket deletion to complete.... [2019-11-28 11:17:00,763] - [rest_client] [140525218207488] - INFO - node 172.23.104.100 existing buckets : [u'travel-sample'] [2019-11-28 11:17:00,764] - [bucket_helper] [140525218207488] - INFO - remove bucket travel-sample ... [2019-11-28 11:17:02,579] - [bucket_helper] [140525218207488] - INFO - deleted bucket : travel-sample from 172.23.104.100 [2019-11-28 11:17:02,579] - [bucket_helper] [140525218207488] - INFO - waiting for bucket deletion to complete.... [2019-11-28 11:17:02,592] - [rest_client] [140525218207488] - INFO - node 172.23.104.100 existing buckets : [] [2019-11-28 11:17:02,592] - [bucket_helper] [140525218207488] - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket', u'travel-sample']) were deleted completely. [2019-11-28 11:17:04,772] - [cluster_helper] [140525218207488] - INFO - rebalancing all nodes in order to remove nodes [2019-11-28 11:17:04,786] - [rest_client] [140525218207488] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.103', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.100,ns_1@172.23.104.103'} [2019-11-28 11:17:04,805] - [rest_client] [140525218207488] - INFO - rebalance operation started [2019-11-28 11:17:04,820] - [rest_client] [140525218207488] - INFO - rebalance percentage : 0.00 % [2019-11-28 11:17:14,854] - [rest_client] [140525218207488] - INFO - rebalance percentage : 50.00 % [2019-11-28 11:17:24,884] - [rest_client] [140525218207488] - INFO - rebalance percentage : 75.00 % [2019-11-28 11:17:44,935] - [rest_client] [140525218207488] - INFO - rebalance progress took 40.13 seconds [2019-11-28 11:17:44,936] - [rest_client] [140525218207488] - INFO - sleep for 10 seconds after rebalance... [2019-11-28 11:17:55,025] - [cluster_helper] [140525218207488] - INFO - removed all the nodes from cluster associated with ip:172.23.104.100 port:8091 ssh_username:root ? [(u'ns_1@172.23.104.107', 8091), (u'ns_1@172.23.104.105', 8091), (u'ns_1@172.23.104.103', 8091)] [2019-11-28 11:17:55,052] - [cluster_helper] [140525218207488] - INFO - waiting for ns_server @ 172.23.104.100:8091 [2019-11-28 11:17:55,067] - [cluster_helper] [140525218207488] - INFO - ns_server @ 172.23.104.100:8091 is running [2019-11-28 11:17:55,080] - [cluster_helper] [140525218207488] - INFO - waiting for ns_server @ 172.23.104.103:8091 [2019-11-28 11:17:55,093] - [cluster_helper] [140525218207488] - INFO - ns_server @ 172.23.104.103:8091 is running [2019-11-28 11:17:55,106] - [cluster_helper] [140525218207488] - INFO - waiting for ns_server @ 172.23.104.105:8091 [2019-11-28 11:17:55,118] - [cluster_helper] [140525218207488] - INFO - ns_server @ 172.23.104.105:8091 is running [2019-11-28 11:17:55,133] - [cluster_helper] [140525218207488] - INFO - waiting for ns_server @ 172.23.104.107:8091 [2019-11-28 11:17:55,157] - [cluster_helper] [140525218207488] - INFO - ns_server @ 172.23.104.107:8091 is running [2019-11-28 11:17:55,158] - [basetestcase] [140525218207488] - INFO - initializing cluster [2019-11-28 11:17:55,160] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:17:55,295] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:17:56,084] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:17:56,220] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:17:57,121] - [remote_util] [140525218207488] - INFO - 172.23.104.100 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:17:57,529] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:17:57,529] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: systemctl stop couchbase-server.service [2019-11-28 11:17:59,857] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:17:59,859] - [basetestcase] [140525218207488] - INFO - Couchbase stopped [2019-11-28 11:17:59,975] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-11-28 11:18:00,012] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:00,012] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-11-28 11:18:00,079] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:00,082] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:18:00,218] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:18:01,026] - [remote_util] [140525218207488] - INFO - 172.23.104.100 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:01,215] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:01,216] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: systemctl start couchbase-server.service [2019-11-28 11:18:01,254] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:01,255] - [basetestcase] [140525218207488] - INFO - Couchbase started [2019-11-28 11:18:01,373] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:18:01,509] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.103 [2019-11-28 11:18:02,474] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:18:02,612] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.103 [2019-11-28 11:18:03,643] - [remote_util] [140525218207488] - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:03,838] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:03,839] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: systemctl stop couchbase-server.service [2019-11-28 11:18:05,413] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:05,414] - [basetestcase] [140525218207488] - INFO - Couchbase stopped [2019-11-28 11:18:05,528] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-11-28 11:18:05,549] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:05,549] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-11-28 11:18:05,633] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:05,636] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:18:05,772] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.103 [2019-11-28 11:18:06,684] - [remote_util] [140525218207488] - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:06,956] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:06,956] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: systemctl start couchbase-server.service [2019-11-28 11:18:06,993] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:06,993] - [basetestcase] [140525218207488] - INFO - Couchbase started [2019-11-28 11:18:07,110] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.105 with username:root [2019-11-28 11:18:07,211] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.105 [2019-11-28 11:18:08,123] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.105 with username:root [2019-11-28 11:18:08,223] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.105 [2019-11-28 11:18:09,167] - [remote_util] [140525218207488] - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:09,421] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:09,422] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.105: systemctl stop couchbase-server.service [2019-11-28 11:18:11,097] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:11,097] - [basetestcase] [140525218207488] - INFO - Couchbase stopped [2019-11-28 11:18:11,212] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-11-28 11:18:11,233] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:11,233] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-11-28 11:18:11,316] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:11,319] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.105 with username:root [2019-11-28 11:18:11,454] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.105 [2019-11-28 11:18:12,402] - [remote_util] [140525218207488] - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:12,567] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:12,567] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.105: systemctl start couchbase-server.service [2019-11-28 11:18:12,689] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:12,690] - [basetestcase] [140525218207488] - INFO - Couchbase started [2019-11-28 11:18:12,758] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.107 with username:root [2019-11-28 11:18:12,926] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.107 [2019-11-28 11:18:14,325] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.107 with username:root [2019-11-28 11:18:14,495] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.107 [2019-11-28 11:18:16,070] - [remote_util] [140525218207488] - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:16,349] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:16,350] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.107: systemctl stop couchbase-server.service [2019-11-28 11:18:18,278] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:18,279] - [basetestcase] [140525218207488] - INFO - Couchbase stopped [2019-11-28 11:18:18,394] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.107: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-11-28 11:18:18,469] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:18,469] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.107: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-11-28 11:18:18,601] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:18,612] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.107 with username:root [2019-11-28 11:18:18,780] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.107 [2019-11-28 11:18:20,433] - [remote_util] [140525218207488] - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists [2019-11-28 11:18:20,774] - [remote_util] [140525218207488] - INFO - Running systemd command on this server [2019-11-28 11:18:20,775] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.107: systemctl start couchbase-server.service [2019-11-28 11:18:20,895] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:18:20,895] - [basetestcase] [140525218207488] - INFO - Couchbase started [2019-11-28 11:18:20,960] - [basetestcase] [140525218207488] - INFO - sleep for 10 secs. ... [2019-11-28 11:18:31,245] - [task] [140518335948544] - INFO - server: ip:172.23.104.100 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.100', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.100', 'uptime': u'26', 'mcdMemoryReserved': 3349, 'storageTotalRam': 4187, 'hostname': u'172.23.104.100:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4897-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3489259520, 'memoryTotal': 4390723584, 'memoryQuota': 1846, 'mcdMemoryAllocated': 3349, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-11-28 11:18:31,257] - [rest_client] [140518335948544] - ERROR - GET http://172.23.104.100:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-11-28 11:18:31,258] - [rest_client] [140518335948544] - INFO - pools/default params : memoryQuota=2243 [2019-11-28 11:18:31,276] - [rest_client] [140518335948544] - INFO - /node/controller/setupServices params on 172.23.104.100:8091:services=kv&password=password&user=Administrator&hostname=172.23.104.100 [2019-11-28 11:18:31,296] - [rest_client] [140518335948544] - INFO - settings/web params on 172.23.104.100:8091:username=Administrator&password=password&port=8091 [2019-11-28 11:18:31,416] - [remote_util] [140518335948544] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:18:31,549] - [remote_util] [140518335948544] - INFO - Connected to 172.23.104.100 [2019-11-28 11:18:32,412] - [remote_util] [140518335948544] - INFO - running command.raw on 172.23.104.100: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-11-28 11:18:32,482] - [remote_util] [140518335948544] - INFO - command executed successfully [2019-11-28 11:18:32,494] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.100:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:32,506] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.100:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:32,507] - [rest_client] [140518335948544] - INFO - settings/indexes params : storageMode=plasma [2019-11-28 11:18:32,607] - [task] [140518335948544] - INFO - server: ip:172.23.104.103 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.103', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.103', 'uptime': u'18', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4024, 'hostname': u'172.23.104.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4897-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3641069568, 'memoryTotal': 4220149760, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-11-28 11:18:32,620] - [rest_client] [140518335948544] - ERROR - GET http://172.23.104.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-11-28 11:18:32,621] - [rest_client] [140518335948544] - INFO - pools/default params : memoryQuota=2156 [2019-11-28 11:18:32,649] - [rest_client] [140518335948544] - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 [2019-11-28 11:18:32,706] - [remote_util] [140518335948544] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:18:32,837] - [remote_util] [140518335948544] - INFO - Connected to 172.23.104.103 [2019-11-28 11:18:33,703] - [remote_util] [140518335948544] - INFO - running command.raw on 172.23.104.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-11-28 11:18:33,753] - [remote_util] [140518335948544] - INFO - command executed successfully [2019-11-28 11:18:33,764] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.103:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:33,775] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.103:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:33,775] - [rest_client] [140518335948544] - INFO - settings/indexes params : storageMode=plasma [2019-11-28 11:18:33,852] - [task] [140518335948544] - INFO - server: ip:172.23.104.105 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.105', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.105', 'uptime': u'18', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.105:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4897-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3678683136, 'memoryTotal': 4220854272, 'memoryQuota': 1749, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-11-28 11:18:33,864] - [rest_client] [140518335948544] - ERROR - GET http://172.23.104.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-11-28 11:18:33,865] - [rest_client] [140518335948544] - INFO - pools/default params : memoryQuota=2157 [2019-11-28 11:18:33,879] - [rest_client] [140518335948544] - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 [2019-11-28 11:18:33,939] - [remote_util] [140518335948544] - INFO - connecting to 172.23.104.105 with username:root [2019-11-28 11:18:34,076] - [remote_util] [140518335948544] - INFO - Connected to 172.23.104.105 [2019-11-28 11:18:34,884] - [remote_util] [140518335948544] - INFO - running command.raw on 172.23.104.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-11-28 11:18:34,936] - [remote_util] [140518335948544] - INFO - command executed successfully [2019-11-28 11:18:34,955] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:34,967] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:34,968] - [rest_client] [140518335948544] - INFO - settings/indexes params : storageMode=plasma [2019-11-28 11:18:35,043] - [task] [140518335948544] - INFO - server: ip:172.23.104.107 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.107', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.107', 'uptime': u'11', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.107:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4897-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3643916288, 'memoryTotal': 4220854272, 'memoryQuota': 1749, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-11-28 11:18:35,060] - [rest_client] [140518335948544] - ERROR - GET http://172.23.104.107:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-11-28 11:18:35,061] - [rest_client] [140518335948544] - INFO - pools/default params : memoryQuota=2157 [2019-11-28 11:18:35,081] - [rest_client] [140518335948544] - INFO - settings/web params on 172.23.104.107:8091:username=Administrator&password=password&port=8091 [2019-11-28 11:18:35,181] - [remote_util] [140518335948544] - INFO - connecting to 172.23.104.107 with username:root [2019-11-28 11:18:35,352] - [remote_util] [140518335948544] - INFO - Connected to 172.23.104.107 [2019-11-28 11:18:36,787] - [remote_util] [140518335948544] - INFO - running command.raw on 172.23.104.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-11-28 11:18:36,909] - [remote_util] [140518335948544] - INFO - command executed successfully [2019-11-28 11:18:36,921] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:36,934] - [rest_client] [140518335948544] - INFO - /diag/eval status on 172.23.104.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-11-28 11:18:36,935] - [rest_client] [140518335948544] - INFO - settings/indexes params : storageMode=plasma [2019-11-28 11:18:37,019] - [basetestcase] [140525218207488] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.100 **** [2019-11-28 11:18:37,083] - [rest_client] [140525218207488] - ERROR - DELETE http://172.23.104.100:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-11-28 11:18:37,084] - [internal_user] [140525218207488] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-11-28 11:18:37,228] - [basetestcase] [140525218207488] - INFO - sleep for 5 secs. ... [2019-11-28 11:18:42,233] - [basetestcase] [140525218207488] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2019-11-28 11:18:42,291] - [basetestcase] [140525218207488] - INFO - done initializing cluster [2019-11-28 11:18:42,996] - [task] [140518335948544] - INFO - adding node 172.23.104.103:8091 to cluster [2019-11-28 11:18:42,996] - [rest_client] [140518335948544] - INFO - adding remote node @172.23.104.103:8091 to this cluster @172.23.104.100:8091 [2019-11-28 11:18:45,126] - [task] [140518335948544] - INFO - adding node 172.23.104.105:8091 to cluster [2019-11-28 11:18:45,126] - [rest_client] [140518335948544] - INFO - adding remote node @172.23.104.105:8091 to this cluster @172.23.104.100:8091 [2019-11-28 11:18:47,307] - [task] [140518335948544] - INFO - adding node 172.23.104.107:8091 to cluster [2019-11-28 11:18:47,307] - [rest_client] [140518335948544] - INFO - adding remote node @172.23.104.107:8091 to this cluster @172.23.104.100:8091 [2019-11-28 11:18:50,920] - [rest_client] [140518335948544] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.105,ns_1@172.23.104.100,ns_1@172.23.104.103'} [2019-11-28 11:18:50,939] - [rest_client] [140518335948544] - INFO - rebalance operation started [2019-11-28 11:18:50,951] - [rest_client] [140518335948544] - INFO - rebalance percentage : 0.00 % [2019-11-28 11:18:50,951] - [task] [140518335948544] - INFO - Rebalance - status: running, progress: 0.00% [2019-11-28 11:19:00,994] - [task] [140518335948544] - INFO - Rebalance - status: none, progress: 100.00% [2019-11-28 11:19:01,021] - [task] [140518335948544] - INFO - rebalancing was completed with progress: 100% in 10.0812001228 sec [2019-11-28 11:19:01,023] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:19:01,159] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:19:01,866] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-11-28 11:19:01,938] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:19:01,938] - [basetestcase] [140525218207488] - INFO - Enabled diag/eval for non-local hosts from 172.23.104.100 [2019-11-28 11:19:01,939] - [basetestcase] [140525218207488] - INFO - ============== basetestcase setup was finished for test #17 test_n1ql_timeout ============== [2019-11-28 11:19:02,005] - [basetestcase] [140525218207488] - INFO - ------- Cluster statistics ------- [2019-11-28 11:19:02,008] - [basetestcase] [140525218207488] - INFO - 172.23.104.107:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.00250626566416, 'mem_free': 3605524480, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'n1ql']} [2019-11-28 11:19:02,009] - [basetestcase] [140525218207488] - INFO - 172.23.104.105:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.758793969849246, 'mem_free': 3622318080, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'index']} [2019-11-28 11:19:02,009] - [basetestcase] [140525218207488] - INFO - 172.23.104.100:8091 => {'swap_mem_used': 0, 'cpu_utilization': 6.459948320413437, 'mem_free': 3460976640, 'swap_mem_total': 0, 'mem_total': 4390723584, 'services': [u'kv']} [2019-11-28 11:19:02,009] - [basetestcase] [140525218207488] - INFO - 172.23.104.103:8091 => {'swap_mem_used': 0, 'cpu_utilization': 14.50381679389313, 'mem_free': 3593883648, 'swap_mem_total': 0, 'mem_total': 4220149760, 'services': [u'eventing']} [2019-11-28 11:19:02,010] - [basetestcase] [140525218207488] - INFO - --- End of cluster statistics --- [2019-11-28 11:19:02,017] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:19:02,152] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:19:41,088] - [basetestcase] [140525218207488] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-11-28 11:19:41,121] - [basetestcase] [140525218207488] - INFO - list of eventing nodes in cluster: [ip:172.23.104.103 port:8091 ssh_username:root] [2019-11-28 11:19:41,137] - [rest_client] [140525218207488] - INFO - settings/indexes params : storageMode=plasma [2019-11-28 11:19:41,153] - [eventing_base] [140525218207488] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2019-11-28 11:19:41,154] - [rest_client] [140525218207488] - INFO - pools/default params : memoryQuota=330 [2019-11-28 11:19:41,172] - [rest_client] [140525218207488] - INFO - pools/default params : indexMemoryQuota=256 [2019-11-28 11:19:41,193] - [rest_client] [140525218207488] - INFO - pools/default params : eventingMemoryQuota=512 [2019-11-28 11:19:41,247] - [basetestcase] [140525218207488] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-11-28 11:19:41,248] - [rest_client] [140525218207488] - INFO - pools/default params : memoryQuota=1200 [2019-11-28 11:19:41,265] - [eventing_n1ql] [140525218207488] - INFO - 200 [2019-11-28 11:19:41,502] - [rest_client] [140518335948544] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=200 [2019-11-28 11:19:41,522] - [rest_client] [140518335948544] - INFO - 0.02 seconds to create bucket src_bucket [2019-11-28 11:19:41,522] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:42,440] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:42,767] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:42,970] - [task] [140518335948544] - WARNING - vbucket map not ready after try 0 [2019-11-28 11:19:42,972] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:43,308] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:44,373] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:44,514] - [task] [140518335948544] - WARNING - vbucket map not ready after try 1 [2019-11-28 11:19:44,515] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:44,777] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:45,073] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:45,215] - [task] [140518335948544] - WARNING - vbucket map not ready after try 2 [2019-11-28 11:19:45,216] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:45,498] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:45,783] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:45,938] - [task] [140518335948544] - WARNING - vbucket map not ready after try 3 [2019-11-28 11:19:45,938] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:46,225] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:47,252] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:47,420] - [task] [140518335948544] - WARNING - vbucket map not ready after try 4 [2019-11-28 11:19:47,421] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:47,702] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:48,014] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 src_bucket [2019-11-28 11:19:48,196] - [task] [140518335948544] - WARNING - vbucket map not ready after try 5 [2019-11-28 11:19:49,287] - [rest_client] [140518335948544] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=200 [2019-11-28 11:19:49,312] - [rest_client] [140518335948544] - INFO - 0.02 seconds to create bucket dst_bucket [2019-11-28 11:19:49,313] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:49,739] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:50,048] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:50,992] - [task] [140518335948544] - WARNING - vbucket map not ready after try 0 [2019-11-28 11:19:50,993] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:51,267] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:51,559] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:51,735] - [task] [140518335948544] - WARNING - vbucket map not ready after try 1 [2019-11-28 11:19:51,736] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:52,066] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:52,412] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:52,571] - [task] [140518335948544] - WARNING - vbucket map not ready after try 2 [2019-11-28 11:19:52,572] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:53,628] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:53,921] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:54,089] - [task] [140518335948544] - WARNING - vbucket map not ready after try 3 [2019-11-28 11:19:54,090] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:54,388] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:54,695] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:54,867] - [task] [140518335948544] - WARNING - vbucket map not ready after try 4 [2019-11-28 11:19:54,868] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.100 to accept set ops [2019-11-28 11:19:55,165] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:55,466] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 dst_bucket [2019-11-28 11:19:56,400] - [task] [140518335948544] - WARNING - vbucket map not ready after try 5 [2019-11-28 11:19:57,501] - [rest_client] [140518335948544] - INFO - http://172.23.104.100:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=200 [2019-11-28 11:19:57,523] - [rest_client] [140518335948544] - INFO - 0.02 seconds to create bucket metadata [2019-11-28 11:19:57,523] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:19:58,454] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:19:58,774] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:19:58,933] - [task] [140518335948544] - WARNING - vbucket map not ready after try 0 [2019-11-28 11:19:58,934] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:19:59,284] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:19:59,568] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:19:59,736] - [task] [140518335948544] - WARNING - vbucket map not ready after try 1 [2019-11-28 11:19:59,737] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:20:00,030] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:01,106] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:01,275] - [task] [140518335948544] - WARNING - vbucket map not ready after try 2 [2019-11-28 11:20:01,275] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:20:01,557] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:01,854] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:02,040] - [task] [140518335948544] - WARNING - vbucket map not ready after try 3 [2019-11-28 11:20:02,040] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:20:02,321] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:02,613] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:03,551] - [task] [140518335948544] - WARNING - vbucket map not ready after try 4 [2019-11-28 11:20:03,552] - [bucket_helper] [140518335948544] - INFO - waiting for memcached bucket : metadata in 172.23.104.100 to accept set ops [2019-11-28 11:20:03,843] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:04,121] - [data_helper] [140518335948544] - INFO - creating direct client 172.23.104.100:11210 metadata [2019-11-28 11:20:04,283] - [task] [140518335948544] - WARNING - vbucket map not ready after try 5 [2019-11-28 11:20:04,658] - [basetestcase] [140525218207488] - INFO - list of n1ql nodes in cluster: [ip:172.23.104.107 port:8091 ssh_username:root] [2019-11-28 11:20:04,659] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:04,679] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:05,143] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 458.07137ms [2019-11-28 11:20:05,144] - [tuq_helper] [140525218207488] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2019-11-28 11:20:05,163] - [rest_client] [140525218207488] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2019-11-28 11:20:08,415] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 3.245757194s [2019-11-28 11:20:08,417] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:08,445] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:08,530] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 64.427659ms [2019-11-28 11:20:08,530] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:08,549] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:08,610] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 55.611121ms [2019-11-28 11:20:08,611] - [tuq_helper] [140525218207488] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2019-11-28 11:20:08,637] - [rest_client] [140525218207488] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2019-11-28 11:20:11,670] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 3.026573223s [2019-11-28 11:20:11,670] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:11,687] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:11,768] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 75.12619ms [2019-11-28 11:20:11,769] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:11,785] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:11,839] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 46.250604ms [2019-11-28 11:20:11,839] - [tuq_helper] [140525218207488] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2019-11-28 11:20:11,856] - [rest_client] [140525218207488] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2019-11-28 11:20:14,918] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 3.055078215s [2019-11-28 11:20:14,918] - [tuq_helper] [140525218207488] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-11-28 11:20:14,950] - [rest_client] [140525218207488] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-11-28 11:20:15,014] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 58.726124ms [2019-11-28 11:20:15,017] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:20:15,221] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.100 [2019-11-28 11:20:16,001] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.100: curl -v -u Administrator:password -X POST http://172.23.104.100:8091/sampleBuckets/install -d '["travel-sample"]' [2019-11-28 11:20:16,157] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:20:16,222] - [basetestcase] [140525218207488] - INFO - sleep for 20 secs. ... [2019-11-28 11:20:36,256] - [eventing_base] [140525218207488] - INFO - Deploying the following handler code : Function_161085069_test_n1ql_timeout with {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} [2019-11-28 11:20:36,257] - [eventing_base] [140525218207488] - INFO - function OnUpdate(doc, meta) { log('docId', meta.id); try{ var i=0; var query = SELECT * From `travel-sample`; for(var q of query){ dst_bucket[i] = q; i++; } query.close(); } catch(e){ log(e); } } function OnDelete(meta) { log('docId', meta.id); try{ var i=0; var query = SELECT * From `travel-sample`; for(var q of query){ delete dst_bucket[i]; i++; } query.close(); } catch(e){ log(e); } } [2019-11-28 11:20:37,818] - [eventing_base] [140525218207488] - INFO - deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_161085069_test_n1ql_timeout' in metakv", "warnings": null } } [2019-11-28 11:20:37,819] - [basetestcase] [140525218207488] - INFO - sleep for 20 secs. Waiting for Function_161085069_test_n1ql_timeout to deployed... ... [2019-11-28 11:20:58,157] - [basetestcase] [140525218207488] - INFO - sleep for 20 secs. Waiting for Function_161085069_test_n1ql_timeout to deployed... ... [2019-11-28 11:21:18,612] - [basetestcase] [140525218207488] - INFO - sleep for 20 secs. Waiting for Function_161085069_test_n1ql_timeout to deployed... ... [2019-11-28 11:21:38,903] - [tuq_helper] [140525218207488] - INFO - RUN QUERY insert into src_bucket (KEY, VALUE) VALUES ("11:21:38.903546","doc created") [2019-11-28 11:21:38,931] - [rest_client] [140525218207488] - INFO - query params : statement=insert+into+src_bucket+%28KEY%2C+VALUE%29+VALUES+%28%2211%3A21%3A38.903546%22%2C%22doc+created%22%29 [2019-11-28 11:21:38,949] - [tuq_helper] [140525218207488] - INFO - TOTAL ELAPSED TIME: 10.753985ms [2019-11-28 11:21:38,950] - [basetestcase] [140525218207488] - INFO - sleep for 30 secs. ... [2019-11-28 11:22:09,003] - [eventing_n1ql] [140525218207488] - INFO - Stats [ { "dcp_feed_boundary": "from_now", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 2240, "agg_queue_memory": 0, "agg_queue_memory_cap": 267386880, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 1, "dcp_mutation_sent_to_worker": 1, "dcp_snapshot": 1, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 240, "failure_stats": 240, "latency_stats": 240, "lcb_exception_stats": 240, "log_level": 3, "num_processed_events": 1, "processed_events_size": 240, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 1, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 1, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 2228, "num_processed_events": 1, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 1, "processed_events_size": 240, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "21816": "2019-11-28T19:22:08Z", "21821": "2019-11-28T19:22:08Z", "21834": "2019-11-28T19:22:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 1, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "21816": "2019-11-28T19:22:08Z", "21821": "2019-11-28T19:22:08Z", "21834": "2019-11-28T19:22:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1507650305, "function_name": "Function_161085069_test_n1ql_timeout", "gocb_creds_request_counter": 8, "internal_vb_distribution_stats": { "worker_Function_161085069_test_n1ql_timeout_0": "[0-341]", "worker_Function_161085069_test_n1ql_timeout_1": "[342-682]", "worker_Function_161085069_test_n1ql_timeout_2": "[683-1023]" }, "latency_percentile_stats": { "100": 9999900, "50": 9999900, "80": 9999900, "90": 9999900, "95": 9999900, "99": 9999900 }, "lcb_creds_request_counter": 20, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.104.103:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.103:8096": { "worker_Function_161085069_test_n1ql_timeout_0": "[0-341]", "worker_Function_161085069_test_n1ql_timeout_1": "[342-682]", "worker_Function_161085069_test_n1ql_timeout_2": "[683-1023]" } }, "worker_pids": { "worker_Function_161085069_test_n1ql_timeout_0": 21816, "worker_Function_161085069_test_n1ql_timeout_1": 21821, "worker_Function_161085069_test_n1ql_timeout_2": 21834 } } ] [2019-11-28 11:22:09,097] - [basetestcase] [140525218207488] - INFO - list of eventing nodes in cluster: [ip:172.23.104.103 port:8091 ssh_username:root] [2019-11-28 11:22:09,103] - [remote_util] [140525218207488] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:22:09,238] - [remote_util] [140525218207488] - INFO - Connected to 172.23.104.103 [2019-11-28 11:22:10,077] - [rest_client] [140525218207488] - INFO - /diag/eval status on 172.23.104.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-11-28 11:22:10,078] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2019-11-28 11:22:10,499] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:22:10,500] - [remote_util] [140525218207488] - INFO - running command.raw on 172.23.104.103: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2019-11-28 11:22:10,534] - [remote_util] [140525218207488] - INFO - command executed successfully [2019-11-28 11:22:10,910] - [eventing_base] [140525218207488] - INFO - Bucket dst_bucket DGM is 100 [2019-11-28 11:22:10,975] - [eventing_base] [140525218207488] - INFO - Bucket metadata DGM is 100 [2019-11-28 11:22:11,047] - [eventing_base] [140525218207488] - INFO - Bucket src_bucket DGM is 100 [2019-11-28 11:22:11,123] - [eventing_base] [140525218207488] - INFO - Bucket travel-sample DGM is 100 [2019-11-28 11:22:11,173] - [basetestcase] [140525218207488] - INFO - ------- Cluster statistics ------- [2019-11-28 11:22:11,174] - [basetestcase] [140525218207488] - INFO - 172.23.104.107:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.503759398496241, 'mem_free': 3533152256, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'n1ql']} [2019-11-28 11:22:11,174] - [basetestcase] [140525218207488] - INFO - 172.23.104.105:8091 => {'swap_mem_used': 0, 'cpu_utilization': 12.80653950953679, 'mem_free': 3220885504, 'swap_mem_total': 3758092288, 'mem_total': 4220854272, 'services': [u'index']} [2019-11-28 11:22:11,175] - [basetestcase] [140525218207488] - INFO - 172.23.104.100:8091 => {'swap_mem_used': 0, 'cpu_utilization': 17.61658031088083, 'mem_free': 2421936128, 'swap_mem_total': 0, 'mem_total': 4390723584, 'services': [u'kv']} [2019-11-28 11:22:11,176] - [basetestcase] [140525218207488] - INFO - 172.23.104.103:8091 => {'swap_mem_used': 0, 'cpu_utilization': 10.57934508816121, 'mem_free': 3347681280, 'swap_mem_total': 0, 'mem_total': 4220149760, 'services': [u'eventing']} [2019-11-28 11:22:11,176] - [basetestcase] [140525218207488] - INFO - --- End of cluster statistics --- [2019-11-28 11:22:11,176] - [basetestcase] [140525218207488] - WARNING - CLEANUP WAS SKIPPED [2019-11-28 11:22:11,224] - [remote_util] [140525455636288] - INFO - connecting to 172.23.104.100 with username:root [2019-11-28 11:22:11,359] - [remote_util] [140525455636288] - INFO - Connected to 172.23.104.100 [2019-11-28 11:22:12,104] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.100: /opt/couchbase/bin/cbcollect_info 172.23.104.100-20191128-1122-diag.zip [2019-11-28 11:39:49,167] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:41:02,492] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.100: rm -f /root/172.23.104.100-20191128-1122-diag.zip [2019-11-28 11:41:02,927] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:41:02,995] - [remote_util] [140525455636288] - INFO - connecting to 172.23.104.103 with username:root [2019-11-28 11:41:03,096] - [remote_util] [140525455636288] - INFO - Connected to 172.23.104.103 [2019-11-28 11:41:03,932] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.103: /opt/couchbase/bin/cbcollect_info 172.23.104.103-20191128-1141-diag.zip [2019-11-28 11:42:45,712] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:42:49,440] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.103: rm -f /root/172.23.104.103-20191128-1141-diag.zip [2019-11-28 11:42:49,528] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:42:49,534] - [remote_util] [140525455636288] - INFO - connecting to 172.23.104.105 with username:root [2019-11-28 11:42:49,670] - [remote_util] [140525455636288] - INFO - Connected to 172.23.104.105 [2019-11-28 11:42:50,534] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.105: /opt/couchbase/bin/cbcollect_info 172.23.104.105-20191128-1142-diag.zip [2019-11-28 11:44:44,692] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:44:45,604] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.105: rm -f /root/172.23.104.105-20191128-1142-diag.zip [2019-11-28 11:44:45,689] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:44:45,806] - [remote_util] [140525455636288] - INFO - connecting to 172.23.104.107 with username:root [2019-11-28 11:44:45,975] - [remote_util] [140525455636288] - INFO - Connected to 172.23.104.107 [2019-11-28 11:44:47,553] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.107: /opt/couchbase/bin/cbcollect_info 172.23.104.107-20191128-1144-diag.zip [2019-11-28 11:46:26,858] - [remote_util] [140525455636288] - INFO - command executed successfully [2019-11-28 11:46:27,643] - [remote_util] [140525455636288] - INFO - running command.raw on 172.23.104.107: rm -f /root/172.23.104.107-20191128-1144-diag.zip [2019-11-28 11:46:27,777] - [remote_util] [140525455636288] - INFO - command executed successfully