./testrunner -i /tmp/recoverytests-add-index.ini doc-per-day=50,standard_buckets=1,sasl_buckets=1,skip_build_tuq=True,cbq_version=sherlock,scan_consistency=request_plus -t 2i.recovery_2i.SecondaryIndexingRecoveryTests.test_rebalance_in,initial=,before=create_index,after=query_ops,groups=simple,dataset=default,doc-per-day=10,nodes_init=4,nodes_in=2,services_in=kv-kv,services_init=n1ql:kv-kv-index-index,GROUP=REB-IN;P1 Test Input params: {'doc-per-day': '50', 'nodes_in': '2', 'GROUP': 'REB-IN;P1', 'services_in': 'kv-kv', 'after': 'query_ops', 'skip_build_tuq': 'True', 'dataset': 'default', 'ini': '/tmp/recoverytests-add-index.ini', 'groups': 'simple', 'case_number': 3, 'sasl_buckets': '1', 'logs_folder': '/data/workspace/w012R2-2i-vset03-03-recoverytests-add-index/logs/testrunner-15-Jul-16_18-49-49/test_3', 'nodes_init': '4', 'before': 'create_index', 'initial': '', 'conf_file': '/tmp/tmp_add_rec.conf', 'num_nodes': 6, 'cluster_name': 'recoverytests-add-index', 'standard_buckets': '1', 'services_init': 'n1ql:kv-kv-index-index', 'scan_consistency': 'request_plus', 'spec': 'tmp_add_rec', 'cbq_version': 'sherlock'} [2015-07-16 21:30:37,577] - [rest_client:1562] INFO - Node versions in cluster [u'4.0.0-3448-enterprise'] [2015-07-16 21:30:37,577] - [basetestcase:128] INFO - ============== basetestcase setup was started for test #3 test_rebalance_in============== [2015-07-16 21:30:37,610] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.191 [2015-07-16 21:30:37,625] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.240 [2015-07-16 21:30:37,639] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.144 [2015-07-16 21:30:37,653] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.145 [2015-07-16 21:30:37,668] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.91 [2015-07-16 21:30:37,683] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.88 [2015-07-16 21:30:37,714] - [basetestcase:309] INFO - sleep for 10 secs. ... [2015-07-16 21:30:47,733] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.191:8091 [2015-07-16 21:30:47,739] - [cluster_helper:80] INFO - ns_server @ 172.23.105.191:8091 is running [2015-07-16 21:30:47,745] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.240:8091 [2015-07-16 21:30:47,751] - [cluster_helper:80] INFO - ns_server @ 172.23.105.240:8091 is running [2015-07-16 21:30:47,759] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.144:8091 [2015-07-16 21:30:47,766] - [cluster_helper:80] INFO - ns_server @ 172.23.107.144:8091 is running [2015-07-16 21:30:47,773] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.145:8091 [2015-07-16 21:30:47,782] - [cluster_helper:80] INFO - ns_server @ 172.23.107.145:8091 is running [2015-07-16 21:30:47,788] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.91:8091 [2015-07-16 21:30:47,793] - [cluster_helper:80] INFO - ns_server @ 172.23.105.91:8091 is running [2015-07-16 21:30:47,799] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.88:8091 [2015-07-16 21:30:47,804] - [cluster_helper:80] INFO - ns_server @ 172.23.105.88:8091 is running [2015-07-16 21:30:47,804] - [basetestcase:147] INFO - initializing cluster [2015-07-16 21:30:48,591] - [rest_client:763] INFO - /node/controller/setupServices params on 172.23.105.191: 8091:services=n1ql%2Ckv&password=password&hostname=172.23.105.191&user=Administrator [2015-07-16 21:30:48,597] - [rest_client:731] ERROR - http://172.23.105.191:8091//node/controller/setupServices error 400 reason: unknown ["cannot change node services after cluster is provisioned"] [2015-07-16 21:30:48,597] - [rest_client:768] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2015-07-16 21:30:48,597] - [rest_client:748] INFO - settings/web params on 172.23.105.191:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,656] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,681] - [rest_client:748] INFO - settings/web params on 172.23.105.240:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,715] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,735] - [rest_client:748] INFO - settings/web params on 172.23.107.144:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,757] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,782] - [rest_client:748] INFO - settings/web params on 172.23.107.145:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,835] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,858] - [rest_client:748] INFO - settings/web params on 172.23.105.91:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,920] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,941] - [rest_client:748] INFO - settings/web params on 172.23.105.88:8091:username=Administrator&password=password&port=8091 [2015-07-16 21:30:48,968] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-16 21:30:48,978] - [basetestcase:159] INFO - done initializing cluster [2015-07-16 21:30:49,980] - [task:370] INFO - adding node 172.23.105.240:8091 to cluster [2015-07-16 21:30:49,980] - [rest_client:948] INFO - adding remote node @172.23.105.240:8091 to this cluster @172.23.105.191:8091 [2015-07-16 21:31:00,367] - [task:370] INFO - adding node 172.23.107.144:8091 to cluster [2015-07-16 21:31:00,367] - [rest_client:948] INFO - adding remote node @172.23.107.144:8091 to this cluster @172.23.105.191:8091 [2015-07-16 21:31:14,587] - [task:370] INFO - adding node 172.23.107.145:8091 to cluster [2015-07-16 21:31:14,588] - [rest_client:948] INFO - adding remote node @172.23.107.145:8091 to this cluster @172.23.105.191:8091 [2015-07-16 21:31:28,872] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.240%2Cns_1%40172.23.107.145%2Cns_1%40172.23.107.144 [2015-07-16 21:31:28,885] - [rest_client:1170] INFO - rebalance operation started [2015-07-16 21:31:28,901] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-16 21:31:38,921] - [task:486] INFO - rebalancing was completed with progress: 100% in 10.0351679325 sec [2015-07-16 21:31:39,951] - [rest_client:1698] INFO - http://172.23.105.191:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=726&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2015-07-16 21:31:39,972] - [rest_client:1720] INFO - 0.02 seconds to create bucket default [2015-07-16 21:31:39,973] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.105.191 to accept set ops [2015-07-16 21:31:41,276] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-16 21:31:41,439] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 default [2015-07-16 21:31:41,706] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-16 21:31:41,864] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 default [2015-07-16 21:31:43,836] - [task:254] INFO - bucket 'default' was created with per node RAM quota: 726 [2015-07-16 21:31:44,890] - [rest_client:1698] INFO - http://172.23.105.191:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=726&proxyPort=11211&authType=sasl&name=bucket0&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=password [2015-07-16 21:31:44,918] - [rest_client:1720] INFO - 0.03 seconds to create bucket bucket0 [2015-07-16 21:31:44,919] - [bucket_helper:305] INFO - waiting for memcached bucket : bucket0 in 172.23.105.191 to accept set ops [2015-07-16 21:31:58,600] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-16 21:31:59,272] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 bucket0 [2015-07-16 21:31:59,933] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-16 21:32:00,111] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 bucket0 [2015-07-16 21:32:01,741] - [task:254] INFO - bucket 'bucket0' was created with per node RAM quota: 726 [2015-07-16 21:32:02,771] - [rest_client:1698] INFO - http://172.23.105.191:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=726 [2015-07-16 21:32:02,793] - [rest_client:1720] INFO - 0.02 seconds to create bucket standard_bucket0 [2015-07-16 21:32:02,794] - [bucket_helper:305] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.191 to accept set ops [2015-07-16 21:32:04,077] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-16 21:32:04,255] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 standard_bucket0 [2015-07-16 21:32:04,542] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-16 21:32:04,698] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 standard_bucket0 [2015-07-16 21:32:06,157] - [task:254] INFO - bucket 'standard_bucket0' was created with per node RAM quota: 726 [2015-07-16 21:32:06,190] - [basetestcase:208] INFO - ============== basetestcase setup was finished for test #3 test_rebalance_in ============== [2015-07-16 21:32:06,234] - [newtuq:34] INFO - Initial status of 172.23.105.191 cluster is warmup [2015-07-16 21:32:06,235] - [newtuq:36] INFO - Waiting for cluster to become healthy [2015-07-16 21:32:06,235] - [basetestcase:309] INFO - sleep for 5 secs. ... [2015-07-16 21:32:11,254] - [newtuq:36] INFO - Waiting for cluster to become healthy [2015-07-16 21:32:11,255] - [basetestcase:309] INFO - sleep for 5 secs. ... [2015-07-16 21:32:16,273] - [newtuq:36] INFO - Waiting for cluster to become healthy [2015-07-16 21:32:16,273] - [basetestcase:309] INFO - sleep for 5 secs. ... [2015-07-16 21:32:21,302] - [newtuq:39] INFO - current status of 172.23.105.191 is healthy [2015-07-16 21:32:21,329] - [newtuq:34] INFO - Initial status of 172.23.105.240 cluster is healthy [2015-07-16 21:32:21,329] - [newtuq:39] INFO - current status of 172.23.105.240 is healthy [2015-07-16 21:32:21,355] - [newtuq:34] INFO - Initial status of 172.23.107.144 cluster is healthy [2015-07-16 21:32:21,356] - [newtuq:39] INFO - current status of 172.23.107.144 is healthy [2015-07-16 21:32:21,380] - [newtuq:34] INFO - Initial status of 172.23.107.145 cluster is healthy [2015-07-16 21:32:21,381] - [newtuq:39] INFO - current status of 172.23.107.145 is healthy [2015-07-16 21:32:21,398] - [newtuq:34] INFO - Initial status of 172.23.105.91 cluster is healthy [2015-07-16 21:32:21,398] - [newtuq:39] INFO - current status of 172.23.105.91 is healthy [2015-07-16 21:32:21,414] - [newtuq:34] INFO - Initial status of 172.23.105.88 cluster is healthy [2015-07-16 21:32:21,414] - [newtuq:39] INFO - current status of 172.23.105.88 is healthy [2015-07-16 21:32:21,415] - [remote_util:158] INFO - connecting to 172.23.105.191 with username : Administrator password : Membase123 ssh_key: [2015-07-16 21:32:26,309] - [remote_util:191] INFO - Connected to 172.23.105.191 [2015-07-16 21:32:56,984] - [basetestcase:1797] INFO - create 100800 to bucket0 documents... [2015-07-16 21:32:57,101] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-16 21:32:57,226] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 bucket0 [2015-07-16 21:32:57,335] - [basetestcase:1797] INFO - create 100800 to default documents... [2015-07-16 21:32:57,438] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-16 21:32:57,572] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 default [2015-07-16 21:32:57,674] - [basetestcase:1797] INFO - create 100800 to standard_bucket0 documents... [2015-07-16 21:32:57,811] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-16 21:32:58,103] - [data_helper:295] INFO - creating direct client 172.23.105.240:11210 standard_bucket0 [2015-07-16 21:46:09,174] - [basetestcase:1808] INFO - LOAD IS FINISHED [2015-07-16 21:46:09,265] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:Administrator] [2015-07-16 21:46:09,265] - [newtuq:84] INFO - ip:172.23.105.191 port:8091 ssh_username:Administrator [2015-07-16 21:46:09,266] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON bucket0 USING VIEW [2015-07-16 21:46:09,266] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:46:09,276] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:46:09,514] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 233.0042ms [2015-07-16 21:46:09,515] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON bucket0 USING VIEW [2015-07-16 21:46:09,525] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+bucket0++USING+VIEW+ [2015-07-16 21:46:56,977] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 47.4479949s [2015-07-16 21:46:56,978] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:46:57,112] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:46:57,631] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 492.0029ms [2015-07-16 21:46:57,632] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON default USING VIEW [2015-07-16 21:46:57,633] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:46:57,662] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:46:58,127] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 445.0265ms [2015-07-16 21:46:58,128] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON default USING VIEW [2015-07-16 21:46:58,165] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default++USING+VIEW+ [2015-07-16 21:47:24,160] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 25.9840012s [2015-07-16 21:47:24,160] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:47:24,194] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:47:24,884] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 679.9665ms [2015-07-16 21:47:24,884] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-16 21:47:24,885] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:47:24,935] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:47:26,323] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.3500009s [2015-07-16 21:47:26,323] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-16 21:47:26,351] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++USING+VIEW+ [2015-07-16 21:48:25,379] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 59.0199977s [2015-07-16 21:48:25,380] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:48:25,391] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:48:25,673] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 275.9984ms [2015-07-16 21:48:25,677] - [base_2i:45] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-16 21:48:25,724] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:Administrator] [2015-07-16 21:48:25,749] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.107.145 port:8091 ssh_username:Administrator, ip:172.23.107.144 port:8091 ssh_username:Administrator] [2015-07-16 21:48:25,750] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:48:25,761] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:48:26,045] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 262.9897ms [2015-07-16 21:48:26,046] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_1) USING GSI [2015-07-16 21:48:26,979] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_1) USING GSI [2015-07-16 21:48:26,989] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_1%29+USING+GSI [2015-07-16 21:48:27,273] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 278.9692ms [2015-07-16 21:48:27,274] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:48:27,286] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:48:27,544] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 252.0023ms [2015-07-16 21:48:37,555] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:48:37,567] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:48:37,859] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 286.0056ms [2015-07-16 21:48:47,870] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:48:47,883] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:48:48,221] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.0063ms [2015-07-16 21:48:48,222] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:48:48,233] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:48:48,527] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 289.0124ms [2015-07-16 21:48:48,528] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_2) USING GSI [2015-07-16 21:48:49,300] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_2) USING GSI [2015-07-16 21:48:49,311] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_2%29+USING+GSI [2015-07-16 21:48:49,830] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 514.0055ms [2015-07-16 21:48:49,831] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:48:50,007] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:48:50,471] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 456.9971ms [2015-07-16 21:49:00,481] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:00,496] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:00,840] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 336.0257ms [2015-07-16 21:49:10,851] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:10,863] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:11,149] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 280.9599ms [2015-07-16 21:49:11,150] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:49:11,161] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:49:11,363] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 196.0222ms [2015-07-16 21:49:11,363] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_3) USING GSI [2015-07-16 21:49:11,856] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_3) USING GSI [2015-07-16 21:49:11,868] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_3%29+USING+GSI [2015-07-16 21:49:12,001] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 126.9963ms [2015-07-16 21:49:12,002] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:12,014] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:12,274] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 246.999ms [2015-07-16 21:49:22,284] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:22,297] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:22,621] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 318.0409ms [2015-07-16 21:49:32,632] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:32,642] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:32,884] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 237.0268ms [2015-07-16 21:49:42,895] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:42,905] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:43,205] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 287.9907ms [2015-07-16 21:49:43,205] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:49:43,219] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:49:43,387] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 162.0126ms [2015-07-16 21:49:43,388] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_4) USING GSI [2015-07-16 21:49:44,039] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_4) USING GSI [2015-07-16 21:49:44,052] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_4%29+USING+GSI [2015-07-16 21:49:44,167] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 107.9745ms [2015-07-16 21:49:44,168] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:44,180] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:44,502] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 304.0482ms [2015-07-16 21:49:54,512] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:49:54,523] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:49:54,752] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 221.9725ms [2015-07-16 21:50:04,763] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:04,774] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:05,101] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 321.0566ms [2015-07-16 21:50:15,112] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:15,123] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:15,422] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 291.9438ms [2015-07-16 21:50:15,422] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:50:15,433] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:50:15,613] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 170.0002ms [2015-07-16 21:50:15,614] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_5) USING GSI [2015-07-16 21:50:16,204] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_5) USING GSI [2015-07-16 21:50:16,216] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_5%29+USING+GSI [2015-07-16 21:50:16,337] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 115.0053ms [2015-07-16 21:50:16,338] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:16,349] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:16,680] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 323.9558ms [2015-07-16 21:50:26,686] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:26,702] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:27,045] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.0426ms [2015-07-16 21:50:37,056] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:37,070] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:37,363] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 282.9892ms [2015-07-16 21:50:37,364] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:50:37,381] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:50:37,576] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 189.0139ms [2015-07-16 21:50:37,577] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_6) USING GSI [2015-07-16 21:50:38,362] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_6) USING GSI [2015-07-16 21:50:38,374] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_6%29+USING+GSI [2015-07-16 21:50:38,494] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 114.9649ms [2015-07-16 21:50:38,495] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:38,506] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:38,808] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 297.0069ms [2015-07-16 21:50:48,818] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:48,831] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:49,109] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 272.9948ms [2015-07-16 21:50:59,120] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:50:59,131] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:50:59,364] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 225.9755ms [2015-07-16 21:51:09,375] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:09,387] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:09,675] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 281.019ms [2015-07-16 21:51:09,676] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:51:09,687] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:51:09,832] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 139.9708ms [2015-07-16 21:51:09,833] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_7) USING GSI [2015-07-16 21:51:10,535] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_7) USING GSI [2015-07-16 21:51:10,546] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_7%29+USING+GSI [2015-07-16 21:51:10,629] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 75.9718ms [2015-07-16 21:51:10,630] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:10,642] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:10,926] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 277.9781ms [2015-07-16 21:51:20,937] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:20,974] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:21,597] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 592.0491ms [2015-07-16 21:51:31,604] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:31,618] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:31,898] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 274.0085ms [2015-07-16 21:51:41,909] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:41,920] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:42,244] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 317.9977ms [2015-07-16 21:51:42,245] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:51:42,256] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:51:42,407] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 145.1403ms [2015-07-16 21:51:42,408] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_8) USING GSI [2015-07-16 21:51:42,666] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_8) USING GSI [2015-07-16 21:51:42,679] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_8%29+USING+GSI [2015-07-16 21:51:42,791] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 101.9529ms [2015-07-16 21:51:42,792] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:42,815] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:43,228] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 405.9954ms [2015-07-16 21:51:53,235] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:51:53,247] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:51:53,498] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 246.0132ms [2015-07-16 21:52:03,509] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:03,521] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:03,806] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 276.991ms [2015-07-16 21:52:13,816] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:13,831] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:14,110] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 271.9918ms [2015-07-16 21:52:14,111] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:52:14,122] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:52:14,281] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 154.0155ms [2015-07-16 21:52:14,281] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_9) USING GSI [2015-07-16 21:52:14,830] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_9) USING GSI [2015-07-16 21:52:14,841] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_9%29+USING+GSI [2015-07-16 21:52:14,926] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 79.9637ms [2015-07-16 21:52:14,927] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:14,940] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:15,244] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 296.0415ms [2015-07-16 21:52:25,255] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:25,269] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:25,598] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 321.0045ms [2015-07-16 21:52:35,609] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:35,622] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:36,050] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 420.0146ms [2015-07-16 21:52:36,051] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:52:36,061] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:52:36,250] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 175.2441ms [2015-07-16 21:52:36,251] - [base_2i:95] INFO - BUILD INDEX on default(index_name_1) USING GSI [2015-07-16 21:52:36,952] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_1) USING GSI [2015-07-16 21:52:37,015] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_1%29+USING+GSI [2015-07-16 21:52:37,225] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 185.8932ms [2015-07-16 21:52:37,226] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:37,237] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:37,620] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 376.0406ms [2015-07-16 21:52:47,631] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:47,645] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:57,473] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 9.8199777s [2015-07-16 21:52:57,474] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:52:57,746] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:52:57,903] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 151.1098ms [2015-07-16 21:52:57,904] - [base_2i:95] INFO - BUILD INDEX on default(index_name_2) USING GSI [2015-07-16 21:52:58,251] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_2) USING GSI [2015-07-16 21:52:58,316] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_2%29+USING+GSI [2015-07-16 21:52:58,409] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 79.0507ms [2015-07-16 21:52:58,410] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:52:58,427] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:52:58,836] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 392.8974ms [2015-07-16 21:53:08,848] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:08,879] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:09,346] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 451.0481ms [2015-07-16 21:53:19,357] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:19,370] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:19,683] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 283.0361ms [2015-07-16 21:53:19,684] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:53:19,727] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:53:19,925] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 160.0306ms [2015-07-16 21:53:19,926] - [base_2i:95] INFO - BUILD INDEX on default(index_name_3) USING GSI [2015-07-16 21:53:20,437] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_3) USING GSI [2015-07-16 21:53:20,450] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_3%29+USING+GSI [2015-07-16 21:53:20,528] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 71.9785ms [2015-07-16 21:53:20,529] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:20,542] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:20,826] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 275.9922ms [2015-07-16 21:53:30,838] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:30,851] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:31,148] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 290.9467ms [2015-07-16 21:53:41,150] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:41,162] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:41,507] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 335.9876ms [2015-07-16 21:53:41,508] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:53:41,519] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:53:41,688] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 163.0165ms [2015-07-16 21:53:41,688] - [base_2i:95] INFO - BUILD INDEX on default(index_name_4) USING GSI [2015-07-16 21:53:42,553] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_4) USING GSI [2015-07-16 21:53:42,566] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_4%29+USING+GSI [2015-07-16 21:53:42,689] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 118.0322ms [2015-07-16 21:53:42,690] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:42,700] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:43,017] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 309.9957ms [2015-07-16 21:53:53,028] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:53:53,043] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:53:53,336] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 283.6461ms [2015-07-16 21:54:03,340] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:03,354] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:03,638] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 277.0055ms [2015-07-16 21:54:13,649] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:13,661] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:13,952] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 286.007ms [2015-07-16 21:54:13,953] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:54:13,964] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:54:14,157] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 187.008ms [2015-07-16 21:54:14,158] - [base_2i:95] INFO - BUILD INDEX on default(index_name_5) USING GSI [2015-07-16 21:54:14,728] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_5) USING GSI [2015-07-16 21:54:14,742] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_5%29+USING+GSI [2015-07-16 21:54:14,834] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 86.0438ms [2015-07-16 21:54:14,835] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:14,846] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:15,129] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 275.9996ms [2015-07-16 21:54:25,140] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:25,172] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:26,224] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 970.9923ms [2015-07-16 21:54:36,236] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:36,250] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:36,517] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 259.0132ms [2015-07-16 21:54:46,528] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:46,540] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:47,005] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 460.0103ms [2015-07-16 21:54:47,006] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:54:47,019] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:54:47,183] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 157.999ms [2015-07-16 21:54:47,183] - [base_2i:95] INFO - BUILD INDEX on default(index_name_6) USING GSI [2015-07-16 21:54:47,871] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_6) USING GSI [2015-07-16 21:54:47,885] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_6%29+USING+GSI [2015-07-16 21:54:47,993] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 100.0135ms [2015-07-16 21:54:47,994] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:48,005] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:48,290] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 275.9975ms [2015-07-16 21:54:58,301] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:54:58,313] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:54:58,584] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 265.0491ms [2015-07-16 21:55:08,595] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:08,608] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:08,994] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 378.0393ms [2015-07-16 21:55:19,005] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:19,018] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:19,300] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 276.0381ms [2015-07-16 21:55:19,301] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:55:19,312] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:55:19,477] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 159.9941ms [2015-07-16 21:55:19,477] - [base_2i:95] INFO - BUILD INDEX on default(index_name_7) USING GSI [2015-07-16 21:55:20,031] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_7) USING GSI [2015-07-16 21:55:20,042] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_7%29+USING+GSI [2015-07-16 21:55:20,124] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 77.0525ms [2015-07-16 21:55:20,125] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:20,137] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:20,477] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 334.0014ms [2015-07-16 21:55:30,481] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:30,492] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:30,811] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 310.9305ms [2015-07-16 21:55:40,820] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:40,836] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:41,207] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 357.9582ms [2015-07-16 21:55:51,210] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:51,302] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:52,643] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.3340115s [2015-07-16 21:55:52,643] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:55:52,654] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:55:52,801] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 141.0527ms [2015-07-16 21:55:52,801] - [base_2i:95] INFO - BUILD INDEX on default(index_name_8) USING GSI [2015-07-16 21:55:53,162] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_8) USING GSI [2015-07-16 21:55:53,173] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_8%29+USING+GSI [2015-07-16 21:55:53,256] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 78.0046ms [2015-07-16 21:55:53,257] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:55:53,268] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:55:53,606] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 329.9992ms [2015-07-16 21:56:03,617] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:03,628] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:03,930] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 296.0182ms [2015-07-16 21:56:13,941] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:13,955] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:14,284] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 322.0034ms [2015-07-16 21:56:24,295] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:24,445] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:25,156] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 704.0556ms [2015-07-16 21:56:25,156] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:56:25,170] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:56:25,340] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 164.0028ms [2015-07-16 21:56:25,340] - [base_2i:95] INFO - BUILD INDEX on default(index_name_9) USING GSI [2015-07-16 21:56:26,296] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_9) USING GSI [2015-07-16 21:56:26,309] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_9%29+USING+GSI [2015-07-16 21:56:26,403] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 86.9845ms [2015-07-16 21:56:26,404] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:26,420] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:27,684] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.2580028s [2015-07-16 21:56:37,695] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:37,705] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:37,999] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 287.9996ms [2015-07-16 21:56:48,010] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:48,025] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:48,302] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 270.0347ms [2015-07-16 21:56:58,305] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:58,317] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:58,656] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 331.9901ms [2015-07-16 21:56:58,657] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:56:58,670] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:56:58,843] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 168.952ms [2015-07-16 21:56:58,844] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_1) USING GSI [2015-07-16 21:56:59,442] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_1) USING GSI [2015-07-16 21:56:59,458] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_1%29+USING+GSI [2015-07-16 21:56:59,635] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 170.0204ms [2015-07-16 21:56:59,636] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:56:59,649] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:56:59,957] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 300.0447ms [2015-07-16 21:57:09,968] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:09,978] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:10,286] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 301.0274ms [2015-07-16 21:57:20,297] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:20,310] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:20,630] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 313.0294ms [2015-07-16 21:57:20,631] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:57:20,640] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:57:20,817] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 170.9975ms [2015-07-16 21:57:20,817] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_2) USING GSI [2015-07-16 21:57:21,660] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_2) USING GSI [2015-07-16 21:57:21,735] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_2%29+USING+GSI [2015-07-16 21:57:21,993] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 234.0449ms [2015-07-16 21:57:21,994] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:22,005] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:22,346] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.984ms [2015-07-16 21:57:32,357] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:32,369] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:32,691] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 314.0098ms [2015-07-16 21:57:42,703] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:42,715] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:47,112] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 4.3879937s [2015-07-16 21:57:47,112] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:57:47,125] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:57:47,263] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 132.9917ms [2015-07-16 21:57:47,264] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_3) USING GSI [2015-07-16 21:57:48,025] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_3) USING GSI [2015-07-16 21:57:48,036] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_3%29+USING+GSI [2015-07-16 21:57:48,122] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 79.9935ms [2015-07-16 21:57:48,123] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:48,135] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:48,527] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 383.9932ms [2015-07-16 21:57:58,538] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:57:58,549] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:57:58,854] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 296.9806ms [2015-07-16 21:58:08,865] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:08,876] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:09,183] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 299.986ms [2015-07-16 21:58:09,183] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:58:09,194] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:58:09,350] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 151.0466ms [2015-07-16 21:58:09,351] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_4) USING GSI [2015-07-16 21:58:10,147] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_4) USING GSI [2015-07-16 21:58:10,161] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_4%29+USING+GSI [2015-07-16 21:58:10,297] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 130.0446ms [2015-07-16 21:58:10,298] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:10,308] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:10,607] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 291.9967ms [2015-07-16 21:58:20,618] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:20,630] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:20,897] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 258.1519ms [2015-07-16 21:58:30,908] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:30,920] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:31,283] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 355.9607ms [2015-07-16 21:58:41,294] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:41,308] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:41,625] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 309.0024ms [2015-07-16 21:58:41,626] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:58:41,638] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:58:41,799] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 154.035ms [2015-07-16 21:58:41,800] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_5) USING GSI [2015-07-16 21:58:42,334] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_5) USING GSI [2015-07-16 21:58:42,347] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_5%29+USING+GSI [2015-07-16 21:58:42,487] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 134.003ms [2015-07-16 21:58:42,488] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:42,499] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:42,747] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 240.9873ms [2015-07-16 21:58:52,758] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:58:52,768] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:58:53,083] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 294.0212ms [2015-07-16 21:59:03,094] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:03,105] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:03,425] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 312.9874ms [2015-07-16 21:59:13,436] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:13,449] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:13,746] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 287.9988ms [2015-07-16 21:59:13,746] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:59:13,758] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:59:13,928] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 163.8573ms [2015-07-16 21:59:13,929] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_6) USING GSI [2015-07-16 21:59:14,524] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_6) USING GSI [2015-07-16 21:59:14,539] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_6%29+USING+GSI [2015-07-16 21:59:14,660] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 116.0003ms [2015-07-16 21:59:14,661] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:14,674] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:14,949] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 268.0027ms [2015-07-16 21:59:24,960] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:24,973] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:25,488] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 437.0315ms [2015-07-16 21:59:35,499] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:35,517] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:35,811] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 284.9863ms [2015-07-16 21:59:45,823] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:45,835] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:46,179] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.9552ms [2015-07-16 21:59:46,180] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 21:59:46,191] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 21:59:46,362] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 162.9308ms [2015-07-16 21:59:46,363] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_7) USING GSI [2015-07-16 21:59:46,697] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_7) USING GSI [2015-07-16 21:59:46,710] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_7%29+USING+GSI [2015-07-16 21:59:46,782] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 68.0621ms [2015-07-16 21:59:46,783] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:46,795] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:47,110] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 309.0021ms [2015-07-16 21:59:57,121] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 21:59:57,134] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 21:59:57,453] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 312.0143ms [2015-07-16 22:00:07,464] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:07,476] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:07,735] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 252.0282ms [2015-07-16 22:00:17,746] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:17,757] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:18,041] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 277.9944ms [2015-07-16 22:00:18,042] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 22:00:18,061] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:00:18,236] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 170.0257ms [2015-07-16 22:00:18,237] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_8) USING GSI [2015-07-16 22:00:18,822] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_8) USING GSI [2015-07-16 22:00:18,837] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_8%29+USING+GSI [2015-07-16 22:00:18,915] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 74.051ms [2015-07-16 22:00:18,916] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:18,928] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:19,236] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 299.9938ms [2015-07-16 22:00:29,241] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:29,253] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:29,999] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 658.9637ms [2015-07-16 22:00:40,010] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:40,021] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:40,409] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 375.0515ms [2015-07-16 22:00:50,420] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:50,433] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:50,706] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 266.0362ms [2015-07-16 22:00:50,707] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-16 22:00:50,719] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:00:51,026] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 301.9926ms [2015-07-16 22:00:51,027] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_9) USING GSI [2015-07-16 22:00:51,954] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_9) USING GSI [2015-07-16 22:00:51,968] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_9%29+USING+GSI [2015-07-16 22:00:52,065] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 90.0348ms [2015-07-16 22:00:52,066] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:00:52,077] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:00:52,382] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 296ms [2015-07-16 22:01:02,393] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:01:02,404] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:01:02,649] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 236.0015ms [2015-07-16 22:01:12,660] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:01:12,673] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:01:12,966] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 285.0503ms [2015-07-16 22:01:12,967] - [recovery_2i:346] INFO - <<<<< START INITIALIZATION PHASE >>>>>> [2015-07-16 22:01:12,967] - [base_2i:729] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-16 22:01:12,967] - [recovery_2i:351] INFO - <<<<< END INITIALIZATION PHASE >>>>>> [2015-07-16 22:01:12,968] - [base_2i:729] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-16 22:01:12,968] - [base_2i:137] INFO - [] [2015-07-16 22:01:12,968] - [basetestcase:309] INFO - sleep for 3 secs. ... [2015-07-16 22:01:13,090] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjoin_yr ON bucket0(join_yr) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:13,101] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjoin_yr+ON+bucket0%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:13,266] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 160.0025ms [2015-07-16 22:01:13,266] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjoin_yr ON default(join_yr) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:13,276] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjoin_yr+ON+default%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:13,475] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 192.9925ms [2015-07-16 22:01:13,475] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjoin_yr ON standard_bucket0(join_yr) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:13,485] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjoin_yr+ON+standard_bucket0%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:13,665] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 174.0495ms [2015-07-16 22:01:15,972] - [basetestcase:309] INFO - sleep for 3 secs. ... [2015-07-16 22:01:16,669] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjob_title ON bucket0(job_title) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:16,680] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjob_title+ON+bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:16,842] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 155.0088ms [2015-07-16 22:01:16,843] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjob_title ON default(job_title) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:16,853] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjob_title+ON+default%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:17,015] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 155.9583ms [2015-07-16 22:01:17,016] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeec287556fe63b48d89632f36bdac4d3ccjob_title ON standard_bucket0(job_title) USING GSI WITH {'defer_build': True} [2015-07-16 22:01:17,030] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employeec287556fe63b48d89632f36bdac4d3ccjob_title+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-16 22:01:17,187] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 149.9996ms [2015-07-16 22:01:18,976] - [base_2i:95] INFO - BUILD INDEX on bucket0(employeec287556fe63b48d89632f36bdac4d3ccjoin_yr,employeec287556fe63b48d89632f36bdac4d3ccjob_title) USING GSI [2015-07-16 22:01:19,191] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(employeec287556fe63b48d89632f36bdac4d3ccjoin_yr,employeec287556fe63b48d89632f36bdac4d3ccjob_title) USING GSI [2015-07-16 22:01:19,203] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28employeec287556fe63b48d89632f36bdac4d3ccjoin_yr%2Cemployeec287556fe63b48d89632f36bdac4d3ccjob_title%29+USING+GSI [2015-07-16 22:01:19,378] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 169.9579ms [2015-07-16 22:01:19,379] - [base_2i:95] INFO - BUILD INDEX on default(employeec287556fe63b48d89632f36bdac4d3ccjoin_yr,employeec287556fe63b48d89632f36bdac4d3ccjob_title) USING GSI [2015-07-16 22:01:20,380] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(employeec287556fe63b48d89632f36bdac4d3ccjoin_yr,employeec287556fe63b48d89632f36bdac4d3ccjob_title) USING GSI [2015-07-16 22:01:20,414] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28employeec287556fe63b48d89632f36bdac4d3ccjoin_yr%2Cemployeec287556fe63b48d89632f36bdac4d3ccjob_title%29+USING+GSI [2015-07-16 22:01:50,881] - [task:2189] ERROR - Unexpected Exception Caught [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 71, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 2190, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Thu Jul 16 22:01:50 2015 [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/2i/recovery_2i.py', 42, 'test_rebalance_in', 'before_index_ops = self._run_before_index_tasks()'), ('pytests/2i/recovery_2i.py', 358, '_run_before_index_tasks', 'scan_consistency = self.scan_consistency, scan_vectors = self.scan_vectors)'), ('pytests/2i/base_2i.py', 425, 'async_check_and_run_operations', 'scan_vectors = scan_vectors)'), ('pytests/2i/base_2i.py', 494, '_async_run_operations', 'scan_consistency = scan_consistency, scan_vectors = scan_vectors)'), ('pytests/2i/base_2i.py', 470, 'async_run_multi_operations', 'tasks += self.async_multi_create_index(buckets, query_definitions)'), ('pytests/2i/base_2i.py', 164, 'async_multi_create_index', 'build_index_task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] [2015-07-16 22:01:50,883] - [base_2i:479] INFO - CBQError: host 172.23.105.191: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'BuildIndexes - cause: Terminate Request due to server termination', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'30.4609927s', u'executionTime': u'30.4609927s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'3c0c31ab-874a-4fa9-93fb-d1dd071295bd', u'signature': None} ERROR [2015-07-16 22:01:50,884] - [recovery_2i:29] INFO - <<<<<< WILL DROP THE INDEXES >>>>> [2015-07-16 22:01:50,995] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:Administrator] [2015-07-16 22:01:50,995] - [tuq_helper:382] INFO - CHECK FOR PRIMARY INDEXES [2015-07-16 22:01:50,995] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON bucket0 USING VIEW [2015-07-16 22:01:50,996] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:01:51,027] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:01:52,841] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.7989841s [2015-07-16 22:01:52,842] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON bucket0 USING VIEW [2015-07-16 22:01:52,852] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+bucket0+USING+VIEW+ [2015-07-16 22:01:52,862] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 6.0008ms [2015-07-16 22:01:52,863] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON default USING VIEW [2015-07-16 22:01:52,863] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:01:52,875] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:01:53,241] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 355.0275ms [2015-07-16 22:01:53,242] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING VIEW [2015-07-16 22:01:53,255] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+VIEW+ [2015-07-16 22:01:53,269] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 7.9744ms [2015-07-16 22:01:53,269] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-16 22:01:53,269] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-16 22:01:53,281] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-16 22:01:53,631] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 338.8841ms [2015-07-16 22:01:53,632] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-16 22:01:53,646] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+standard_bucket0+USING+VIEW+ [2015-07-16 22:01:53,660] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 7.9921ms [2015-07-16 22:01:54,246] - [remote_util:754] INFO - File /cygdrive/c/Windows/win.ini was found [2015-07-16 22:01:54,246] - [remote_util:2108] INFO - This is windows server! [2015-07-16 22:01:54,366] - [remote_util:754] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2015-07-16 22:01:54,471] - [remote_util:795] INFO - get windows information [2015-07-16 22:01:54,487] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: hostname [2015-07-16 22:01:54,654] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:54,656] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: echo "ipconfig" > /tmp/cmd.bat; /tmp/cmd.bat [2015-07-16 22:01:54,934] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:54,936] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: rm -f /opt/couchbase/var/lib/couchbase/meta;rm -f /tmp/log_upr_client.sock [2015-07-16 22:01:55,444] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:55,445] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: killall cbq-engine [2015-07-16 22:01:55,739] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:55,740] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: killall tuqtng [2015-07-16 22:01:55,954] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:55,956] - [remote_util:1919] INFO - running command.raw on 172.23.105.191: killall indexer [2015-07-16 22:01:56,158] - [remote_util:1956] INFO - command executed successfully [2015-07-16 22:01:56,160] - [basetestcase:241] INFO - ============== basetestcase cleanup was started for test #3 test_rebalance_in ============== [2015-07-16 22:01:56,284] - [bucket_helper:137] INFO - deleting existing buckets [u'bucket0', u'default', u'standard_bucket0'] on 172.23.105.191 [2015-07-16 22:01:56,285] - [bucket_helper:139] INFO - remove bucket bucket0 ... [2015-07-16 22:02:01,204] - [bucket_helper:153] INFO - deleted bucket : bucket0 from 172.23.105.191 [2015-07-16 22:02:01,204] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-16 22:02:01,282] - [rest_client:124] INFO - node 172.23.105.191 existing buckets : [u'default', u'standard_bucket0'] [2015-07-16 22:02:01,284] - [bucket_helper:139] INFO - remove bucket default ... [2015-07-16 22:02:06,191] - [bucket_helper:153] INFO - deleted bucket : default from 172.23.105.191 [2015-07-16 22:02:06,192] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-16 22:02:06,222] - [rest_client:124] INFO - node 172.23.105.191 existing buckets : [u'standard_bucket0'] [2015-07-16 22:02:06,224] - [bucket_helper:139] INFO - remove bucket standard_bucket0 ... [2015-07-16 22:02:09,240] - [bucket_helper:153] INFO - deleted bucket : standard_bucket0 from 172.23.105.191 [2015-07-16 22:02:09,241] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-16 22:02:09,247] - [rest_client:124] INFO - node 172.23.105.191 existing buckets : [] [2015-07-16 22:02:09,265] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.240 [2015-07-16 22:02:09,281] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.144 [2015-07-16 22:02:09,296] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.145 [2015-07-16 22:02:09,309] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.91 [2015-07-16 22:02:09,322] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.88 [2015-07-16 22:02:09,359] - [cluster_helper:260] INFO - rebalancing all nodes in order to remove nodes [2015-07-16 22:02:09,365] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.240%2Cns_1%40172.23.107.145%2Cns_1%40172.23.107.144&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.240%2Cns_1%40172.23.107.145%2Cns_1%40172.23.107.144 [2015-07-16 22:02:09,373] - [rest_client:1170] INFO - rebalance operation started [2015-07-16 22:02:09,409] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-16 22:02:19,437] - [rest_client:1243] INFO - rebalance progress took 10.0634999275 seconds [2015-07-16 22:02:19,437] - [rest_client:1244] INFO - sleep for 10 seconds after rebalance... [2015-07-16 22:02:29,486] - [cluster_helper:300] INFO - removed all the nodes from cluster associated with ip:172.23.105.191 port:8091 ssh_username:Administrator ? [(u'ns_1@172.23.105.240', 8091), (u'ns_1@172.23.107.145', 8091), (u'ns_1@172.23.107.144', 8091)] [2015-07-16 22:02:29,487] - [basetestcase:309] INFO - sleep for 10 secs. ... [2015-07-16 22:02:39,505] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.191:8091 [2015-07-16 22:02:39,514] - [cluster_helper:80] INFO - ns_server @ 172.23.105.191:8091 is running [2015-07-16 22:02:39,522] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.240:8091 [2015-07-16 22:02:39,529] - [cluster_helper:80] INFO - ns_server @ 172.23.105.240:8091 is running [2015-07-16 22:02:39,536] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.144:8091 [2015-07-16 22:02:39,542] - [cluster_helper:80] INFO - ns_server @ 172.23.107.144:8091 is running [2015-07-16 22:02:39,549] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.145:8091 [2015-07-16 22:02:39,555] - [cluster_helper:80] INFO - ns_server @ 172.23.107.145:8091 is running [2015-07-16 22:02:39,563] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.91:8091 [2015-07-16 22:02:39,569] - [cluster_helper:80] INFO - ns_server @ 172.23.105.91:8091 is running [2015-07-16 22:02:39,579] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.88:8091 [2015-07-16 22:02:39,588] - [cluster_helper:80] INFO - ns_server @ 172.23.105.88:8091 is running [2015-07-16 22:02:39,589] - [basetestcase:260] INFO - ============== basetestcase cleanup was finished for test #3 test_rebalance_in ============== ====================================================================== ERROR: test_rebalance_in (2i.recovery_2i.SecondaryIndexingRecoveryTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/2i/recovery_2i.py", line 42, in test_rebalance_in before_index_ops = self._run_before_index_tasks() File "pytests/2i/recovery_2i.py", line 358, in _run_before_index_tasks scan_consistency = self.scan_consistency, scan_vectors = self.scan_vectors) File "pytests/2i/base_2i.py", line 425, in async_check_and_run_operations scan_vectors = scan_vectors) File "pytests/2i/base_2i.py", line 494, in _async_run_operations scan_consistency = scan_consistency, scan_vectors = scan_vectors) File "pytests/2i/base_2i.py", line 470, in async_run_multi_operations tasks += self.async_multi_create_index(buckets, query_definitions) File "pytests/2i/base_2i.py", line 164, in async_multi_create_index build_index_task.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception CBQError: CBQError: host 172.23.105.191: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'BuildIndexes - cause: Terminate Request due to server termination', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'30.4609927s', u'executionTime': u'30.4609927s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'3c0c31ab-874a-4fa9-93fb-d1dd071295bd', u'signature': None} ---------------------------------------------------------------------- Ran 1 test in 1922.045s