testrunner > ./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_2i.ini initial_index_number=3,get-cbcollect-info=True -t 2i.memdb_oom_2i.SecondaryIndexMemdbOomTests.test_oom_increase_mem_quota,gsi_type=memory_optimized,nodes_init=4,services_init=kv:n1ql-kv-index-index,doc_ops=True,standard_buckets=2,default_buckets=1,initial_index_number=4,scan_consistency=request_plus,skip_cleanup=True Global Test input params: {'cluster_name': 'dev-4-nodes-xdcr_n1ql_2i', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_2i.ini', 'num_nodes': 4} Logs will be stored at /Users/deepkaran/workspace/cb_watson_r/testrunner/logs/testrunner-16-Aug-11_14-21-22/test_1 ./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_2i.ini -t 2i.memdb_oom_2i.SecondaryIndexMemdbOomTests.test_oom_increase_mem_quota,gsi_type=memory_optimized,nodes_init=4,services_init=kv:n1ql-kv-index-index,doc_ops=True,standard_buckets=2,default_buckets=1,initial_index_number=4,scan_consistency=request_plus,skip_cleanup=True Test Input params: {'cluster_name': 'dev-4-nodes-xdcr_n1ql_2i', 'logs_folder': '/Users/deepkaran/workspace/cb_watson_r/testrunner/logs/testrunner-16-Aug-11_14-21-22/test_1', 'gsi_type': 'memory_optimized', 'doc_ops': 'True', 'standard_buckets': '2', 'default_buckets': '1', 'num_nodes': 4, 'skip_cleanup': 'True', 'services_init': 'kv:n1ql-kv-index-index', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_2i.ini', 'initial_index_number': '4', 'scan_consistency': 'request_plus', 'case_number': 1, 'nodes_init': '4'} Run before suite setup for 2i.memdb_oom_2i.SecondaryIndexMemdbOomTests.test_oom_increase_mem_quota test_oom_increase_mem_quota (2i.memdb_oom_2i.SecondaryIndexMemdbOomTests) ... 2016-08-11 14:21:22 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://127.0.0.1:9000/pools/default error 404 reason: unknown "unknown pool" 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [] 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_oom_increase_mem_quota============== 2016-08-11 14:21:22 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://127.0.0.1:9000/pools/default/ error 404 reason: unknown "unknown pool" 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 127.0.0.1 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 127.0.0.1 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 127.0.0.1 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 127.0.0.1 2016-08-11 14:21:22 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running 2016-08-11 14:21:32 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://127.0.0.1:9000/pools/default error 404 reason: unknown "unknown pool" 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 2016-08-11 14:21:32 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force 2016-08-11 14:21:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4368 2016-08-11 14:21:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://127.0.0.1:9000/pools/default error 400 reason: unknown {"errors":{"_":"Total quota (4368MB) exceeds the maximum allowed quota (3072MB) on node 'n_0@127.0.0.1'"}} 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:services=kv%2Cn1ql&password=asdasd&hostname=127.0.0.1%3A9000&user=Administrator 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9000:username=Administrator&password=asdasd&port=9000 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4368 2016-08-11 14:21:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://127.0.0.1:9001/pools/default error 400 reason: unknown {"errors":{"_":"Total quota (4368MB) exceeds the maximum allowed quota (3072MB) on node 'n_1@127.0.0.1'"}} 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9001:username=Administrator&password=asdasd&port=9001 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4368 2016-08-11 14:21:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://127.0.0.1:9002/pools/default error 400 reason: unknown {"errors":{"_":"Total quota (4368MB) exceeds the maximum allowed quota (3072MB) on node 'n_2@127.0.0.1'"}} 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9002:username=Administrator&password=asdasd&port=9002 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4368 2016-08-11 14:21:33 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://127.0.0.1:9003/pools/default error 400 reason: unknown {"errors":{"_":"Total quota (4368MB) exceeds the maximum allowed quota (3073MB) on node 'n_3@127.0.0.1'"}} 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2016-08-11 14:21:33 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9003:username=Administrator&password=asdasd&port=9003 2016-08-11 14:21:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2016-08-11 14:21:34 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9001 to cluster 2016-08-11 14:21:34 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9001 to this cluster @127.0.0.1:9000 2016-08-11 14:21:37 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9002 to cluster 2016-08-11 14:21:37 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9002 to this cluster @127.0.0.1:9000 2016-08-11 14:21:40 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9003 to cluster 2016-08-11 14:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9003 to this cluster @127.0.0.1:9000 2016-08-11 14:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=asdasd&ejectedNodes=&user=Administrator&knownNodes=n_2%40127.0.0.1%2Cn_3%40127.0.0.1%2Cn_1%40127.0.0.1%2Cn_0%4010.17.6.105 2016-08-11 14:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2016-08-11 14:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2016-08-11 14:21:52 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.026777029 sec 2016-08-11 14:21:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=768&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None 2016-08-11 14:21:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2016-08-11 14:21:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops 2016-08-11 14:21:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 default 2016-08-11 14:21:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 default 2016-08-11 14:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 default 2016-08-11 14:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 default 2016-08-11 14:21:56 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 768 2016-08-11 14:21:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=768 2016-08-11 14:21:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket standard_bucket0 2016-08-11 14:21:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: proxyPort=11218&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=768 2016-08-11 14:21:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket standard_bucket1 2016-08-11 14:21:57 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : standard_bucket0 in 127.0.0.1 to accept set ops 2016-08-11 14:22:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket0 2016-08-11 14:22:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket0 2016-08-11 14:22:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket0 2016-08-11 14:22:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket0 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'standard_bucket0' was created with per node RAM quota: 768 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : standard_bucket1 in 127.0.0.1 to accept set ops 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket1 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket1 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket1 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket1 2016-08-11 14:22:07 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'standard_bucket1' was created with per node RAM quota: 768 2016-08-11 14:22:07 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_oom_increase_mem_quota ============== 2016-08-11 14:22:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9002 ssh_username:Administrator, ip:127.0.0.1 port:9003 ssh_username:Administrator] 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username : Administrator password : asdasd ssh_key: 2016-08-11 14:22:17 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [basetestcase.load] create 296352 to default documents... 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 default 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 default 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [basetestcase.load] create 296352 to standard_bucket0 documents... 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket0 2016-08-11 14:22:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket0 2016-08-11 14:22:33 | INFO | MainProcess | test_thread | [basetestcase.load] create 296352 to standard_bucket1 documents... 2016-08-11 14:22:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket1 2016-08-11 14:22:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket1 2016-08-11 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2016-08-11 14:27:04 | INFO | MainProcess | test_thread | [newtuq.setUp] {'remaining': {'start': 0, 'end': 49}} 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator] 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator number of buckets 3 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON default USING VIEW 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 173.924243ms 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default USING VIEW 2016-08-11 14:27:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default++USING+VIEW+ 2016-08-11 14:27:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 14.145792965s 2016-08-11 14:27:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:27:33 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 150.643748ms 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 135.82638ms 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW 2016-08-11 14:27:34 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++USING+VIEW+ 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.644071726s 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 112.128319ms 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON standard_bucket1 USING VIEW 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:27:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:27:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 107.055996ms 2016-08-11 14:27:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON standard_bucket1 USING VIEW 2016-08-11 14:27:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket1++USING+VIEW+ 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 12.118435377s 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 192.051897ms 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [base_2i.setUp] {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'before': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}} 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator] 2016-08-11 14:28:00 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9002 ssh_username:Administrator, ip:127.0.0.1 port:9003 ssh_username:Administrator] 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_0 ON default(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_0+ON+default%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 147.250825ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_1 ON default(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_1+ON+default%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 63.858576ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_2 ON default(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_2+ON+default%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 67.409876ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_3 ON default(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_3+ON+default%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 69.712099ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_0 ON standard_bucket0(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_0+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 67.006006ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_1 ON standard_bucket0(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_1+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 64.684536ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_2 ON standard_bucket0(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_2+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 87.83203ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_3 ON standard_bucket0(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_3+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.796825ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_0 ON standard_bucket1(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_0+ON+standard_bucket1%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.801316ms 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_1 ON standard_bucket1(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:01 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_1+ON+standard_bucket1%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 81.915128ms 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_2 ON standard_bucket1(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_2+ON+standard_bucket1%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 63.838653ms 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX index_name_3 ON standard_bucket1(job_title) USING GSI WITH {'nodes': ['127.0.0.1:9002'], 'defer_build': True} 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+index_name_3+ON+standard_bucket1%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27127.0.0.1%3A9002%27%5D%2C+%27defer_build%27%3A+True%7D 2016-08-11 14:28:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 69.101416ms 2016-08-11 14:28:02 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on default(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:02 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on standard_bucket0(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:02 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on standard_bucket1(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:03 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:03 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28index_name_0%2Cindex_name_1%2Cindex_name_2%2Cindex_name_3%29+USING+GSI 2016-08-11 14:28:03 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 95.064636ms 2016-08-11 14:28:03 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on standard_bucket0(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:03 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_0%2Cindex_name_1%2Cindex_name_2%2Cindex_name_3%29+USING+GSI 2016-08-11 14:28:06 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.481934268s 2016-08-11 14:28:06 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on standard_bucket1(index_name_0,index_name_1,index_name_2,index_name_3) USING GSI 2016-08-11 14:28:06 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+standard_bucket1%28index_name_0%2Cindex_name_1%2Cindex_name_2%2Cindex_name_3%29+USING+GSI 2016-08-11 14:28:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.904618484s 2016-08-11 14:28:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 543.614025ms 2016-08-11 14:28:14 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:14 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:14 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 615.342487ms 2016-08-11 14:28:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:15 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:16 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 463.36307ms 2016-08-11 14:28:17 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:17 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 378.8703ms 2016-08-11 14:28:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 647.18626ms 2016-08-11 14:28:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:20 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 359.730972ms 2016-08-11 14:28:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:21 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 243.327729ms 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'index_name_3' 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_3%27 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 146.945632ms 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [memdb_oom_2i.setUp] Setting indexer memory quota to 300 MB... 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [rest_client.set_indexer_memoryQuota] pools/default params : indexMemoryQuota=300 2016-08-11 14:28:23 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [basetestcase.async_load] create 100800 to default documents... 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 default 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 default 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [basetestcase.async_load] create 100800 to standard_bucket0 documents... 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket0 2016-08-11 14:28:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket0 2016-08-11 14:28:58 | INFO | MainProcess | test_thread | [basetestcase.async_load] create 100800 to standard_bucket1 documents... 2016-08-11 14:28:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.17.6.105:12000 standard_bucket1 2016-08-11 14:28:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12002 standard_bucket1 2016-08-11 14:36:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2016-08-11 14:37:08 | INFO | MainProcess | test_thread | [memdb_oom_2i._push_indexer_off_the_cliff] OOM on 127.0.0.1 is achieved 2016-08-11 14:37:08 | INFO | MainProcess | test_thread | [memdb_oom_2i.test_oom_increase_mem_quota] Current Indexer Memory Quota is 300 2016-08-11 14:37:08 | INFO | MainProcess | test_thread | [memdb_oom_2i.test_oom_increase_mem_quota] Increasing Indexer Memory Quota to 500 2016-08-11 14:37:08 | INFO | MainProcess | test_thread | [rest_client.set_indexer_memoryQuota] pools/default params : indexMemoryQuota=500 2016-08-11 14:37:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 120 secs. ... 2016-08-11 14:39:08 | INFO | MainProcess | test_thread | [memdb_oom_2i.test_oom_increase_mem_quota] Indexer out of OOM... 2016-08-11 14:39:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 180 secs. ... 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9001 ssh_username:Administrator, ip:127.0.0.1 port:9000 ssh_username:Administrator] 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.get_buckets_itemCount] {u'default': 100800, u'standard_bucket0': 100800, u'standard_bucket1': 100800} 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM default USE INDEX (index_name_0) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+default+USE+INDEX+%28index_name_0%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 61.215205ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM default USE INDEX (index_name_1) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+default+USE+INDEX+%28index_name_1%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 12.732481ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM default USE INDEX (index_name_2) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+default+USE+INDEX+%28index_name_2%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 15.308112ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM default USE INDEX (index_name_3) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+default+USE+INDEX+%28index_name_3%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 15.689651ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket0 USE INDEX (index_name_0) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket0+USE+INDEX+%28index_name_0%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 14.144819ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket0 USE INDEX (index_name_1) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket0+USE+INDEX+%28index_name_1%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 14.547724ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket0 USE INDEX (index_name_2) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket0+USE+INDEX+%28index_name_2%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 15.808782ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket0 USE INDEX (index_name_3) 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket0+USE+INDEX+%28index_name_3%29 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 14.542686ms 2016-08-11 14:42:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket1 USE INDEX (index_name_0) 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket1+USE+INDEX+%28index_name_0%29 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 15.788637ms 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket1 USE INDEX (index_name_1) 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket1+USE+INDEX+%28index_name_1%29 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.203991ms 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket1 USE INDEX (index_name_2) 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket1+USE+INDEX+%28index_name_2%29 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.538151ms 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM standard_bucket1 USE INDEX (index_name_3) 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+standard_bucket1+USE+INDEX+%28index_name_3%29 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.231064ms 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [base_2i._verify_bucket_count_with_index_count] Items Indexed Verified with bucket count... 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [base_2i.query_using_index] Query : SELECT job_title FROM default WHERE job_title != "Sales" 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is doc["job_title"] != "Sales" 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None 2016-08-11 14:42:09 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"job_title" : doc["job_title"],} 2016-08-11 14:42:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT job_title FROM default WHERE job_title != "Sales" 2016-08-11 14:42:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+job_title+FROM+default++WHERE++job_title+%21%3D+%22Sales%22+ 2016-08-11 14:42:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.641328994s 2016-08-11 14:42:23 | INFO | MainProcess | test_thread | [tuq_helper._verify_results] Analyzing Actual Result 2016-08-11 14:42:24 | INFO | MainProcess | test_thread | [tuq_helper._verify_results] Analyzing Expected Result 2016-08-11 14:42:24 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT job_title FROM default WHERE job_title != "Sales" 2016-08-11 14:42:24 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+job_title+FROM+default++WHERE++job_title+%21%3D+%22Sales%22+ 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.981738889s 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [tuq_helper._verify_results] Analyzing Actual Result 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [tuq_helper._verify_results] Analyzing Expected Result FAIL 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:127.0.0.1 port:9002 ssh_username:Administrator, ip:127.0.0.1 port:9003 ssh_username:Administrator] 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username : Administrator password : asdasd ssh_key: 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: "/Users/deepkaran/workspace/cb_watson_r/ns_server/logs/n_2" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: /sbin/sysctl -n machdep.cpu.brand_string 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/Users/deepkaran/workspace/cb_watson_r/ns_server/logs/n_2"/indexer.log* | wc -l 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username : Administrator password : asdasd ssh_key: 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: "/Users/deepkaran/workspace/cb_watson_r/ns_server/logs/n_3" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: /sbin/sysctl -n machdep.cpu.brand_string 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/Users/deepkaran/workspace/cb_watson_r/ns_server/logs/n_3"/indexer.log* | wc -l 2016-08-11 14:42:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-08-11 14:42:34 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force ====================================================================== FAIL: test_oom_increase_mem_quota (2i.memdb_oom_2i.SecondaryIndexMemdbOomTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/2i/memdb_oom_2i.py", line 81, in test_oom_increase_mem_quota query_definitions=self.load_query_definitions) File "pytests/2i/base_2i.py", line 348, in multi_query_using_index scan_vector=scan_vector, verify_results=verify_results) File "pytests/2i/base_2i.py", line 295, in query_using_index self.assertTrue(check, msg) AssertionError: Results are incorrect.Actual num 67192. Expected num: 65856. ---------------------------------------------------------------------- Ran 1 test in 1272.229s FAILED (failures=1) summary so far suite 2i.memdb_oom_2i.SecondaryIndexMemdbOomTests , pass 0 , fail 1