./testrunner -i /tmp/recoverytests-add-index.ini doc-per-day=50,standard_buckets=1,sasl_buckets=1,skip_build_tuq=True,cbq_version=sherlock,scan_consistency=request_plus -t 2i.recovery_2i.SecondaryIndexingRecoveryTests.test_rebalance_out,initial=,before=create_index,after=query_ops,groups=simple,dataset=default,doc-per-day=10,services_init=n1ql:kv-kv-kv-index-index,nodes_init=5,nodes_out=1,nodes_out_dist=kv:1,GROUP=REB-OUT;P0 Test Input params: {'doc-per-day': '50', 'standard_buckets': '1', 'GROUP': 'REB-OUT;P0', 'after': 'query_ops', 'skip_build_tuq': 'True', 'dataset': 'default', 'ini': '/tmp/recoverytests-add-index.ini', 'groups': 'simple', 'case_number': 7, 'sasl_buckets': '1', 'logs_folder': '/data/workspace/w012R2-2i-vset03-03-recoverytests-add-index/logs/testrunner-15-Jul-19_21-12-23/test_7', 'nodes_init': '5', 'before': 'create_index', 'initial': '', 'conf_file': '/tmp/tmp_add_rec.conf', 'num_nodes': 6, 'cluster_name': 'recoverytests-add-index', 'services_init': 'n1ql:kv-kv-kv-index-index', 'nodes_out': '1', 'scan_consistency': 'request_plus', 'nodes_out_dist': 'kv:1', 'spec': 'tmp_add_rec', 'cbq_version': 'sherlock'} [2015-07-20 05:23:15,055] - [rest_client:1544] INFO - Node 172.23.107.145 not part of cluster inactiveAdded [2015-07-20 05:23:15,055] - [rest_client:1562] INFO - Node versions in cluster [u'4.0.0-3465-enterprise'] [2015-07-20 05:23:15,055] - [basetestcase:128] INFO - ============== basetestcase setup was started for test #7 test_rebalance_out============== [2015-07-20 05:23:15,091] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.144 [2015-07-20 05:23:15,106] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.145 [2015-07-20 05:23:15,108] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:16,112] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:17,117] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:18,122] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:19,126] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:20,131] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:21,134] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:22,140] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:23,157] - [rest_client:731] ERROR - http://172.23.105.191:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster." http://172.23.105.191:8091/nodes/self with status False: Node is unknown to this cluster. [2015-07-20 05:23:23,158] - [rest_client:272] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2015-07-20 05:23:28,165] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:29,170] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:30,174] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:31,178] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:32,183] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:33,187] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:34,192] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:23:35,229] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.191 [2015-07-20 05:23:35,242] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.40 [2015-07-20 05:23:35,256] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.91 [2015-07-20 05:23:35,269] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.88 [2015-07-20 05:23:35,305] - [cluster_helper:260] INFO - rebalancing all nodes in order to remove nodes [2015-07-20 05:23:35,311] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.107.145&user=Administrator&knownNodes=ns_1%40172.23.107.144%2Cns_1%40172.23.107.145 [2015-07-20 05:23:35,320] - [rest_client:1170] INFO - rebalance operation started [2015-07-20 05:23:35,370] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-20 05:23:45,395] - [rest_client:1243] INFO - rebalance progress took 10.0746638775 seconds [2015-07-20 05:23:45,396] - [rest_client:1244] INFO - sleep for 10 seconds after rebalance... [2015-07-20 05:23:55,421] - [cluster_helper:300] INFO - removed all the nodes from cluster associated with ip:172.23.107.144 port:8091 ssh_username:Administrator ? [(u'ns_1@172.23.107.145', 8091)] [2015-07-20 05:23:55,421] - [basetestcase:309] INFO - sleep for 10 secs. ... [2015-07-20 05:24:05,442] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.144:8091 [2015-07-20 05:24:05,450] - [cluster_helper:80] INFO - ns_server @ 172.23.107.144:8091 is running [2015-07-20 05:24:05,457] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.145:8091 [2015-07-20 05:24:05,465] - [cluster_helper:80] INFO - ns_server @ 172.23.107.145:8091 is running [2015-07-20 05:24:05,468] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:06,472] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:07,476] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:08,480] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:09,485] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:10,489] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:11,493] - [rest_client:734] ERROR - socket error while connecting to http://172.23.105.191:8091/nodes/self error [Errno 111] Connection refused [2015-07-20 05:24:12,502] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.191:8091 [2015-07-20 05:24:12,509] - [cluster_helper:80] INFO - ns_server @ 172.23.105.191:8091 is running [2015-07-20 05:24:12,518] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.40:8091 [2015-07-20 05:24:12,526] - [cluster_helper:80] INFO - ns_server @ 172.23.105.40:8091 is running [2015-07-20 05:24:12,535] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.91:8091 [2015-07-20 05:24:12,546] - [cluster_helper:80] INFO - ns_server @ 172.23.105.91:8091 is running [2015-07-20 05:24:12,555] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.88:8091 [2015-07-20 05:24:12,563] - [cluster_helper:80] INFO - ns_server @ 172.23.105.88:8091 is running [2015-07-20 05:24:12,563] - [basetestcase:147] INFO - initializing cluster [2015-07-20 05:24:13,122] - [rest_client:763] INFO - /node/controller/setupServices params on 172.23.107.144: 8091:services=n1ql%2Ckv&password=password&hostname=172.23.107.144&user=Administrator [2015-07-20 05:24:13,128] - [rest_client:731] ERROR - http://172.23.107.144:8091//node/controller/setupServices error 400 reason: unknown ["cannot change node services after cluster is provisioned"] [2015-07-20 05:24:13,128] - [rest_client:768] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2015-07-20 05:24:13,128] - [rest_client:748] INFO - settings/web params on 172.23.107.144:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,193] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-20 05:24:13,249] - [rest_client:748] INFO - settings/web params on 172.23.107.145:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,278] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-20 05:24:13,312] - [rest_client:748] INFO - settings/web params on 172.23.105.191:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,344] - [rest_client:785] INFO - pools/default params : memoryQuota=0 [2015-07-20 05:24:13,350] - [rest_client:731] ERROR - http://172.23.105.191:8091/pools/default error 400 reason: unknown {"errors":{"memoryQuota":"The RAM Quota value is too small. Quota must be between 256 MB and 3270 MB (80% of memory size)."}} [2015-07-20 05:24:13,368] - [rest_client:748] INFO - settings/web params on 172.23.105.40:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,396] - [rest_client:785] INFO - pools/default params : memoryQuota=2154 [2015-07-20 05:24:13,416] - [rest_client:748] INFO - settings/web params on 172.23.105.91:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,443] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-20 05:24:13,469] - [rest_client:748] INFO - settings/web params on 172.23.105.88:8091:username=Administrator&password=password&port=8091 [2015-07-20 05:24:13,494] - [rest_client:785] INFO - pools/default params : memoryQuota=2180 [2015-07-20 05:24:13,500] - [basetestcase:159] INFO - done initializing cluster [2015-07-20 05:24:14,502] - [task:370] INFO - adding node 172.23.107.145:8091 to cluster [2015-07-20 05:24:14,502] - [rest_client:948] INFO - adding remote node @172.23.107.145:8091 to this cluster @172.23.107.144:8091 [2015-07-20 05:24:22,540] - [task:370] INFO - adding node 172.23.105.191:8091 to cluster [2015-07-20 05:24:22,540] - [rest_client:948] INFO - adding remote node @172.23.105.191:8091 to this cluster @172.23.107.144:8091 [2015-07-20 05:24:38,049] - [task:370] INFO - adding node 172.23.105.40:8091 to cluster [2015-07-20 05:24:38,050] - [rest_client:948] INFO - adding remote node @172.23.105.40:8091 to this cluster @172.23.107.144:8091 [2015-07-20 05:24:45,830] - [task:370] INFO - adding node 172.23.105.91:8091 to cluster [2015-07-20 05:24:45,831] - [rest_client:948] INFO - adding remote node @172.23.105.91:8091 to this cluster @172.23.107.144:8091 [2015-07-20 05:24:53,505] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.144%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91 [2015-07-20 05:24:53,519] - [rest_client:1170] INFO - rebalance operation started [2015-07-20 05:24:53,532] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-20 05:25:03,551] - [task:486] INFO - rebalancing was completed with progress: 100% in 10.0322701931 sec [2015-07-20 05:25:04,582] - [rest_client:1698] INFO - http://172.23.107.144:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=718&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2015-07-20 05:25:04,599] - [rest_client:1720] INFO - 0.02 seconds to create bucket default [2015-07-20 05:25:04,600] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.107.144 to accept set ops [2015-07-20 05:25:06,097] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-20 05:25:06,294] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 default [2015-07-20 05:25:06,586] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 default [2015-07-20 05:25:07,044] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-20 05:25:07,380] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 default [2015-07-20 05:25:07,583] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 default [2015-07-20 05:25:08,935] - [task:254] INFO - bucket 'default' was created with per node RAM quota: 718 [2015-07-20 05:25:09,973] - [rest_client:1698] INFO - http://172.23.107.144:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=718&proxyPort=11211&authType=sasl&name=bucket0&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=password [2015-07-20 05:25:09,994] - [rest_client:1720] INFO - 0.02 seconds to create bucket bucket0 [2015-07-20 05:25:09,994] - [bucket_helper:305] INFO - waiting for memcached bucket : bucket0 in 172.23.107.144 to accept set ops [2015-07-20 05:25:13,616] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-20 05:25:13,782] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 bucket0 [2015-07-20 05:25:13,951] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 bucket0 [2015-07-20 05:25:14,324] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-20 05:25:14,508] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 bucket0 [2015-07-20 05:25:14,759] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 bucket0 [2015-07-20 05:25:15,985] - [task:254] INFO - bucket 'bucket0' was created with per node RAM quota: 718 [2015-07-20 05:25:17,023] - [rest_client:1698] INFO - http://172.23.107.144:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=718 [2015-07-20 05:25:17,046] - [rest_client:1720] INFO - 0.02 seconds to create bucket standard_bucket0 [2015-07-20 05:25:17,047] - [bucket_helper:305] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.107.144 to accept set ops [2015-07-20 05:25:29,916] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-20 05:25:30,333] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 standard_bucket0 [2015-07-20 05:25:30,637] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 standard_bucket0 [2015-07-20 05:25:30,902] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-20 05:25:31,067] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 standard_bucket0 [2015-07-20 05:25:31,373] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 standard_bucket0 [2015-07-20 05:25:32,930] - [task:254] INFO - bucket 'standard_bucket0' was created with per node RAM quota: 718 [2015-07-20 05:25:32,943] - [basetestcase:208] INFO - ============== basetestcase setup was finished for test #7 test_rebalance_out ============== [2015-07-20 05:25:33,007] - [newtuq:34] INFO - Initial status of 172.23.107.144 cluster is warmup [2015-07-20 05:25:33,008] - [newtuq:36] INFO - Waiting for cluster to become healthy [2015-07-20 05:25:33,008] - [basetestcase:309] INFO - sleep for 5 secs. ... [2015-07-20 05:25:38,035] - [newtuq:39] INFO - current status of 172.23.107.144 is healthy [2015-07-20 05:25:38,067] - [newtuq:34] INFO - Initial status of 172.23.107.145 cluster is healthy [2015-07-20 05:25:38,068] - [newtuq:39] INFO - current status of 172.23.107.145 is healthy [2015-07-20 05:25:38,102] - [newtuq:34] INFO - Initial status of 172.23.105.191 cluster is healthy [2015-07-20 05:25:38,102] - [newtuq:39] INFO - current status of 172.23.105.191 is healthy [2015-07-20 05:25:38,133] - [newtuq:34] INFO - Initial status of 172.23.105.40 cluster is healthy [2015-07-20 05:25:38,133] - [newtuq:39] INFO - current status of 172.23.105.40 is healthy [2015-07-20 05:25:38,164] - [newtuq:34] INFO - Initial status of 172.23.105.91 cluster is healthy [2015-07-20 05:25:38,164] - [newtuq:39] INFO - current status of 172.23.105.91 is healthy [2015-07-20 05:25:38,180] - [newtuq:34] INFO - Initial status of 172.23.105.88 cluster is healthy [2015-07-20 05:25:38,181] - [newtuq:39] INFO - current status of 172.23.105.88 is healthy [2015-07-20 05:25:38,181] - [remote_util:158] INFO - connecting to 172.23.107.144 with username : Administrator password : Membase123 ssh_key: [2015-07-20 05:25:43,094] - [remote_util:191] INFO - Connected to 172.23.107.144 [2015-07-20 05:26:10,818] - [basetestcase:1797] INFO - create 100800 to bucket0 documents... [2015-07-20 05:26:11,113] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 bucket0 [2015-07-20 05:26:11,260] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 bucket0 [2015-07-20 05:26:11,411] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 bucket0 [2015-07-20 05:26:11,517] - [basetestcase:1797] INFO - create 100800 to default documents... [2015-07-20 05:26:11,612] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 default [2015-07-20 05:26:11,768] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 default [2015-07-20 05:26:12,067] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 default [2015-07-20 05:26:12,310] - [basetestcase:1797] INFO - create 100800 to standard_bucket0 documents... [2015-07-20 05:26:12,525] - [data_helper:295] INFO - creating direct client 172.23.105.191:11210 standard_bucket0 [2015-07-20 05:26:12,768] - [data_helper:295] INFO - creating direct client 172.23.107.144:11210 standard_bucket0 [2015-07-20 05:26:13,056] - [data_helper:295] INFO - creating direct client 172.23.107.145:11210 standard_bucket0 [2015-07-20 05:36:19,590] - [basetestcase:1808] INFO - LOAD IS FINISHED [2015-07-20 05:36:19,671] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.107.144 port:8091 ssh_username:Administrator] [2015-07-20 05:36:19,671] - [newtuq:84] INFO - ip:172.23.107.144 port:8091 ssh_username:Administrator [2015-07-20 05:36:19,672] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON bucket0 USING VIEW [2015-07-20 05:36:19,672] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:36:19,690] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:36:20,086] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 389.9977ms [2015-07-20 05:36:20,086] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON bucket0 USING VIEW [2015-07-20 05:36:20,103] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+bucket0++USING+VIEW+ [2015-07-20 05:37:19,151] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 59.0349554s [2015-07-20 05:37:19,152] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:37:19,166] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:37:19,444] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 270.9829ms [2015-07-20 05:37:19,445] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON default USING VIEW [2015-07-20 05:37:19,445] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:37:19,457] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:37:19,715] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 252.9398ms [2015-07-20 05:37:19,716] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON default USING VIEW [2015-07-20 05:37:19,728] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default++USING+VIEW+ [2015-07-20 05:37:43,990] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 24.2549575s [2015-07-20 05:37:43,991] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:37:44,003] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:37:44,295] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 284.9998ms [2015-07-20 05:37:44,297] - [tuq_helper:406] INFO - CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-20 05:37:44,297] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:37:44,307] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:37:44,580] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 268.943ms [2015-07-20 05:37:44,581] - [tuq_helper:72] INFO - RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-20 05:37:44,592] - [rest_client:2237] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++USING+VIEW+ [2015-07-20 05:38:14,152] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 29.5539967s [2015-07-20 05:38:14,153] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:14,165] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:14,558] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 387.0344ms [2015-07-20 05:38:14,561] - [base_2i:45] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-20 05:38:14,610] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.107.144 port:8091 ssh_username:Administrator] [2015-07-20 05:38:14,630] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.105.40 port:8091 ssh_username:Administrator, ip:172.23.105.91 port:8091 ssh_username:Administrator] [2015-07-20 05:38:14,631] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:38:14,640] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:38:14,853] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 207.9953ms [2015-07-20 05:38:14,854] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_1) USING GSI [2015-07-20 05:38:15,733] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_1) USING GSI [2015-07-20 05:38:15,745] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_1%29+USING+GSI [2015-07-20 05:38:15,986] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 236.0011ms [2015-07-20 05:38:15,988] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:15,999] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:16,296] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 289.0341ms [2015-07-20 05:38:26,308] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:26,322] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:26,661] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.996ms [2015-07-20 05:38:36,672] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:36,684] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:37,017] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 325.007ms [2015-07-20 05:38:37,017] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:38:37,032] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:38:37,243] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 204.0142ms [2015-07-20 05:38:37,244] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_2) USING GSI [2015-07-20 05:38:38,014] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_2) USING GSI [2015-07-20 05:38:38,027] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_2%29+USING+GSI [2015-07-20 05:38:38,287] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 253.9953ms [2015-07-20 05:38:38,289] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:38,305] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:38,628] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 317.9966ms [2015-07-20 05:38:48,639] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:48,651] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:48,965] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 308.0438ms [2015-07-20 05:38:58,975] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:38:58,989] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:38:59,290] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 293.9989ms [2015-07-20 05:38:59,291] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:38:59,302] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:38:59,516] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 208.9966ms [2015-07-20 05:38:59,517] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_3) USING GSI [2015-07-20 05:39:00,316] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_3) USING GSI [2015-07-20 05:39:00,331] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_3%29+USING+GSI [2015-07-20 05:39:00,438] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 101.0529ms [2015-07-20 05:39:00,439] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:00,451] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:00,789] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 331.9983ms [2015-07-20 05:39:10,800] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:10,814] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:11,159] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 337.9948ms [2015-07-20 05:39:21,170] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:21,183] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:21,539] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 333.0049ms [2015-07-20 05:39:21,540] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:39:21,568] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:39:21,767] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 193.9978ms [2015-07-20 05:39:21,767] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_4) USING GSI [2015-07-20 05:39:22,465] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_4) USING GSI [2015-07-20 05:39:22,478] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_4%29+USING+GSI [2015-07-20 05:39:22,576] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 92.7345ms [2015-07-20 05:39:22,577] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:22,588] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:22,920] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 326.0025ms [2015-07-20 05:39:32,931] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:32,949] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:52,805] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 19.8469964s [2015-07-20 05:39:52,806] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:39:52,818] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:39:53,022] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 196.0472ms [2015-07-20 05:39:53,022] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_5) USING GSI [2015-07-20 05:39:53,615] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_5) USING GSI [2015-07-20 05:39:53,629] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_5%29+USING+GSI [2015-07-20 05:39:53,724] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 87.6261ms [2015-07-20 05:39:53,726] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:39:53,739] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:39:54,194] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 447.0013ms [2015-07-20 05:40:04,206] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:04,219] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:04,541] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 316.9898ms [2015-07-20 05:40:14,552] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:14,572] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:14,965] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 383.002ms [2015-07-20 05:40:14,966] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:40:14,979] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:40:15,167] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 182.0432ms [2015-07-20 05:40:15,167] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_6) USING GSI [2015-07-20 05:40:15,752] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_6) USING GSI [2015-07-20 05:40:15,766] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_6%29+USING+GSI [2015-07-20 05:40:15,861] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 89.9591ms [2015-07-20 05:40:15,862] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:15,876] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:16,301] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 419.0122ms [2015-07-20 05:40:26,312] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:26,331] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:26,853] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 513.0448ms [2015-07-20 05:40:36,864] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:36,877] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:43,255] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 6.3710309s [2015-07-20 05:40:43,256] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:40:43,267] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:40:43,474] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 201.9958ms [2015-07-20 05:40:43,475] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_7) USING GSI [2015-07-20 05:40:43,896] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_7) USING GSI [2015-07-20 05:40:43,910] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_7%29+USING+GSI [2015-07-20 05:40:44,015] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 94.9891ms [2015-07-20 05:40:44,016] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:44,027] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:40:44,473] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 440.9774ms [2015-07-20 05:40:54,484] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:40:54,495] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:04,570] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 10.0689727s [2015-07-20 05:41:04,571] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:41:04,582] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:41:04,774] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 185.9428ms [2015-07-20 05:41:04,775] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_8) USING GSI [2015-07-20 05:41:05,042] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_8) USING GSI [2015-07-20 05:41:05,055] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_8%29+USING+GSI [2015-07-20 05:41:05,195] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 135.9944ms [2015-07-20 05:41:05,197] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:05,208] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:05,507] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 292.001ms [2015-07-20 05:41:15,518] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:15,530] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:15,872] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 334.9969ms [2015-07-20 05:41:25,883] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:25,894] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:26,210] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 310.001ms [2015-07-20 05:41:36,221] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:36,233] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:36,533] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 293.0137ms [2015-07-20 05:41:36,533] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:41:36,545] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:41:36,759] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 207.9776ms [2015-07-20 05:41:36,760] - [base_2i:95] INFO - BUILD INDEX on bucket0(index_name_9) USING GSI [2015-07-20 05:41:37,234] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(index_name_9) USING GSI [2015-07-20 05:41:37,248] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28index_name_9%29+USING+GSI [2015-07-20 05:41:37,347] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 92.9794ms [2015-07-20 05:41:37,348] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:37,361] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:37,637] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 270.003ms [2015-07-20 05:41:47,648] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:47,664] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:47,942] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 272.0013ms [2015-07-20 05:41:57,953] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:41:57,968] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:41:59,054] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.0739822s [2015-07-20 05:42:09,065] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:09,080] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:09,416] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 329.9951ms [2015-07-20 05:42:09,416] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:42:09,428] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:42:09,643] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 209.9543ms [2015-07-20 05:42:09,643] - [base_2i:95] INFO - BUILD INDEX on default(index_name_1) USING GSI [2015-07-20 05:42:10,388] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_1) USING GSI [2015-07-20 05:42:10,403] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_1%29+USING+GSI [2015-07-20 05:42:10,675] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 265.0074ms [2015-07-20 05:42:10,677] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:10,888] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:11,527] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 632.0411ms [2015-07-20 05:42:21,539] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:21,554] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:21,872] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 310.9968ms [2015-07-20 05:42:31,883] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:31,898] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:32,251] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 347.0187ms [2015-07-20 05:42:32,251] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:42:32,263] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:42:32,545] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 272.9705ms [2015-07-20 05:42:32,546] - [base_2i:95] INFO - BUILD INDEX on default(index_name_2) USING GSI [2015-07-20 05:42:32,704] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_2) USING GSI [2015-07-20 05:42:32,717] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_2%29+USING+GSI [2015-07-20 05:42:32,966] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 243.0437ms [2015-07-20 05:42:32,967] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:32,979] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:33,303] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 317.0076ms [2015-07-20 05:42:43,314] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:43,330] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:43,652] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 315.0298ms [2015-07-20 05:42:53,664] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:42:53,681] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:42:54,007] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 319.0297ms [2015-07-20 05:43:04,018] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:04,030] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:04,362] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 324.9934ms [2015-07-20 05:43:14,373] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:14,387] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:14,701] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 309.0013ms [2015-07-20 05:43:14,702] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:43:14,713] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:43:14,907] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 188.9594ms [2015-07-20 05:43:14,908] - [base_2i:95] INFO - BUILD INDEX on default(index_name_3) USING GSI [2015-07-20 05:43:15,018] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_3) USING GSI [2015-07-20 05:43:15,030] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_3%29+USING+GSI [2015-07-20 05:43:15,171] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 134.999ms [2015-07-20 05:43:15,172] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:15,185] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:15,730] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 538.9742ms [2015-07-20 05:43:25,741] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:25,754] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:26,071] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 308.0604ms [2015-07-20 05:43:36,082] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:36,094] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:36,411] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 308.9752ms [2015-07-20 05:43:36,413] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:43:36,425] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:43:36,630] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 197.0038ms [2015-07-20 05:43:36,631] - [base_2i:95] INFO - BUILD INDEX on default(index_name_4) USING GSI [2015-07-20 05:43:37,199] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_4) USING GSI [2015-07-20 05:43:37,212] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_4%29+USING+GSI [2015-07-20 05:43:37,306] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 87.0403ms [2015-07-20 05:43:37,308] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:37,322] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:37,621] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 292.0133ms [2015-07-20 05:43:47,632] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:47,690] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:48,313] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 511.9542ms [2015-07-20 05:43:58,324] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:43:58,631] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:43:59,241] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 601.027ms [2015-07-20 05:43:59,242] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:43:59,257] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:43:59,455] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 192.9933ms [2015-07-20 05:43:59,456] - [base_2i:95] INFO - BUILD INDEX on default(index_name_5) USING GSI [2015-07-20 05:44:00,335] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_5) USING GSI [2015-07-20 05:44:00,350] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_5%29+USING+GSI [2015-07-20 05:44:00,456] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 100.0368ms [2015-07-20 05:44:00,457] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:00,469] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:00,769] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 296.0005ms [2015-07-20 05:44:10,780] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:10,877] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:11,456] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 572.042ms [2015-07-20 05:44:21,467] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:21,481] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:21,811] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 323.0008ms [2015-07-20 05:44:21,811] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:44:21,823] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:44:22,035] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 206.0046ms [2015-07-20 05:44:22,036] - [base_2i:95] INFO - BUILD INDEX on default(index_name_6) USING GSI [2015-07-20 05:44:22,483] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_6) USING GSI [2015-07-20 05:44:22,497] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_6%29+USING+GSI [2015-07-20 05:44:22,600] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 97.008ms [2015-07-20 05:44:22,601] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:22,615] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:22,913] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 291.9863ms [2015-07-20 05:44:32,924] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:32,938] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:33,277] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 333.007ms [2015-07-20 05:44:43,288] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:43,304] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:43,664] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 352.018ms [2015-07-20 05:44:43,665] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:44:43,676] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:44:43,886] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 206.0277ms [2015-07-20 05:44:43,887] - [base_2i:95] INFO - BUILD INDEX on default(index_name_7) USING GSI [2015-07-20 05:44:44,626] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_7) USING GSI [2015-07-20 05:44:44,638] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_7%29+USING+GSI [2015-07-20 05:44:44,751] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 106.0033ms [2015-07-20 05:44:44,752] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:44,765] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:45,094] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 320.9976ms [2015-07-20 05:44:55,105] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:44:55,121] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:44:55,505] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 376.997ms [2015-07-20 05:45:05,516] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:05,527] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:05,901] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 367.9931ms [2015-07-20 05:45:05,902] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:45:05,915] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:45:06,106] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 185.002ms [2015-07-20 05:45:06,106] - [base_2i:95] INFO - BUILD INDEX on default(index_name_8) USING GSI [2015-07-20 05:45:06,779] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_8) USING GSI [2015-07-20 05:45:06,790] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_8%29+USING+GSI [2015-07-20 05:45:06,893] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 97.9944ms [2015-07-20 05:45:06,894] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:06,907] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:07,278] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 362.9993ms [2015-07-20 05:45:17,289] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:17,309] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:17,733] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 417.0451ms [2015-07-20 05:45:27,744] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:28,435] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:29,515] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.0659487s [2015-07-20 05:45:29,516] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON default(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:45:29,530] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+default%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:45:29,730] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 191.0046ms [2015-07-20 05:45:29,730] - [base_2i:95] INFO - BUILD INDEX on default(index_name_9) USING GSI [2015-07-20 05:45:29,923] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(index_name_9) USING GSI [2015-07-20 05:45:29,936] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28index_name_9%29+USING+GSI [2015-07-20 05:45:30,036] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 93.8846ms [2015-07-20 05:45:30,037] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:30,051] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:30,391] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 331.9937ms [2015-07-20 05:45:40,402] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:40,429] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:40,860] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 424.0008ms [2015-07-20 05:45:50,871] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:45:50,927] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:45:51,304] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 368.9809ms [2015-07-20 05:45:51,305] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_1 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:45:51,315] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_1+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:46:01,706] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 10.3860408s [2015-07-20 05:46:01,707] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_1) USING GSI [2015-07-20 05:46:02,075] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_1) USING GSI [2015-07-20 05:46:02,097] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_1%29+USING+GSI [2015-07-20 05:46:02,350] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 246.0175ms [2015-07-20 05:46:02,351] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:02,365] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:02,688] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 316.046ms [2015-07-20 05:46:12,699] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:12,715] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:13,082] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 358.0054ms [2015-07-20 05:46:23,093] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:23,107] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:23,470] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 353.9449ms [2015-07-20 05:46:23,471] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_2 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:46:23,486] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_2+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:46:23,697] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 204.0565ms [2015-07-20 05:46:23,697] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_2) USING GSI [2015-07-20 05:46:24,376] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_2) USING GSI [2015-07-20 05:46:24,390] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_2%29+USING+GSI [2015-07-20 05:46:24,600] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 205.0153ms [2015-07-20 05:46:24,601] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:24,615] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:24,968] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 347.0361ms [2015-07-20 05:46:34,979] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:34,993] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:35,428] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 427.981ms [2015-07-20 05:46:45,439] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:45,456] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:45,965] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 501.0202ms [2015-07-20 05:46:45,966] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_3 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:46:45,980] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_3+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:46:46,228] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 242.0092ms [2015-07-20 05:46:46,228] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_3) USING GSI [2015-07-20 05:46:46,627] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_3) USING GSI [2015-07-20 05:46:46,639] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_3%29+USING+GSI [2015-07-20 05:46:46,770] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 125.9926ms [2015-07-20 05:46:46,771] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:46,783] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:47,108] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 317.9981ms [2015-07-20 05:46:57,119] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:46:57,133] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:46:57,525] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 384.0318ms [2015-07-20 05:47:07,536] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:07,552] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:07,885] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 326.0304ms [2015-07-20 05:47:17,895] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:17,915] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:18,367] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 445.9571ms [2015-07-20 05:47:18,368] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_4 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:47:18,379] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_4+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:47:18,570] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 187.0031ms [2015-07-20 05:47:18,571] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_4) USING GSI [2015-07-20 05:47:18,810] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_4) USING GSI [2015-07-20 05:47:18,822] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_4%29+USING+GSI [2015-07-20 05:47:18,924] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 96.9823ms [2015-07-20 05:47:18,924] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:18,936] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:19,272] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 327.944ms [2015-07-20 05:47:29,283] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:29,507] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:30,119] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 595.0269ms [2015-07-20 05:47:40,130] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:40,142] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:40,478] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 329.9968ms [2015-07-20 05:47:40,478] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_5 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:47:40,488] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_5+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:47:43,438] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 2.9450096s [2015-07-20 05:47:43,439] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_5) USING GSI [2015-07-20 05:47:43,954] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_5) USING GSI [2015-07-20 05:47:43,967] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_5%29+USING+GSI [2015-07-20 05:47:44,062] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 89.0204ms [2015-07-20 05:47:44,064] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:44,075] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:44,411] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 329.9581ms [2015-07-20 05:47:54,422] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:47:54,434] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:47:54,749] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 308.0075ms [2015-07-20 05:48:04,760] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:04,772] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:05,097] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 316.5449ms [2015-07-20 05:48:05,098] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_6 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:48:05,109] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_6+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:48:05,300] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 184.9994ms [2015-07-20 05:48:05,300] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_6) USING GSI [2015-07-20 05:48:06,089] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_6) USING GSI [2015-07-20 05:48:06,101] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_6%29+USING+GSI [2015-07-20 05:48:06,245] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 135.8688ms [2015-07-20 05:48:06,246] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:06,261] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:06,609] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 340.0038ms [2015-07-20 05:48:16,620] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:16,633] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:17,308] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 665.9959ms [2015-07-20 05:48:27,319] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:27,330] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:27,634] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 295.9921ms [2015-07-20 05:48:27,635] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_7 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:48:27,648] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_7+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:48:27,876] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 221.9936ms [2015-07-20 05:48:27,876] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_7) USING GSI [2015-07-20 05:48:28,271] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_7) USING GSI [2015-07-20 05:48:28,287] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_7%29+USING+GSI [2015-07-20 05:48:28,396] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 103.1005ms [2015-07-20 05:48:28,397] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:28,408] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:33,820] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 5.4040492s [2015-07-20 05:48:43,831] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:43,844] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:44,190] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 337.046ms [2015-07-20 05:48:54,202] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:54,213] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:54,539] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 319.0014ms [2015-07-20 05:48:54,540] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_8 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:48:54,551] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_8+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:48:54,747] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 190.9942ms [2015-07-20 05:48:54,748] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_8) USING GSI [2015-07-20 05:48:55,428] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_8) USING GSI [2015-07-20 05:48:55,440] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_8%29+USING+GSI [2015-07-20 05:48:55,609] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 165.0577ms [2015-07-20 05:48:55,610] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:48:55,622] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:48:55,950] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 320.0008ms [2015-07-20 05:49:05,960] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:49:05,974] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:49:06,329] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 343.9974ms [2015-07-20 05:49:16,340] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:49:16,353] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:49:16,720] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 359.0502ms [2015-07-20 05:49:16,720] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX index_name_9 ON standard_bucket0(join_mo) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:16,732] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+index_name_9+ON+standard_bucket0%28join_mo%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:17,236] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 500.0017ms [2015-07-20 05:49:17,236] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(index_name_9) USING GSI [2015-07-20 05:49:17,636] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(index_name_9) USING GSI [2015-07-20 05:49:17,646] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28index_name_9%29+USING+GSI [2015-07-20 05:49:17,805] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 153.0314ms [2015-07-20 05:49:17,806] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:49:17,817] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:49:18,700] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 875.9939ms [2015-07-20 05:49:28,706] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:49:28,790] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:49:29,905] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.1079841s [2015-07-20 05:49:39,917] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:49:39,928] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:49:40,273] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 337.993ms [2015-07-20 05:49:40,274] - [recovery_2i:346] INFO - <<<<< START INITIALIZATION PHASE >>>>>> [2015-07-20 05:49:40,274] - [base_2i:729] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-20 05:49:40,274] - [recovery_2i:351] INFO - <<<<< END INITIALIZATION PHASE >>>>>> [2015-07-20 05:49:40,274] - [base_2i:729] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-20 05:49:40,275] - [base_2i:137] INFO - [] [2015-07-20 05:49:40,275] - [basetestcase:309] INFO - sleep for 3 secs. ... [2015-07-20 05:49:40,832] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjoin_yr ON bucket0(join_yr) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:40,843] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjoin_yr+ON+bucket0%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:41,026] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 178.9961ms [2015-07-20 05:49:41,027] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjoin_yr ON default(join_yr) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:41,038] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjoin_yr+ON+default%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:41,240] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 197.0337ms [2015-07-20 05:49:41,240] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjoin_yr ON standard_bucket0(join_yr) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:41,251] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjoin_yr+ON+standard_bucket0%28join_yr%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:41,457] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 200.9978ms [2015-07-20 05:49:43,279] - [basetestcase:309] INFO - sleep for 3 secs. ... [2015-07-20 05:49:43,460] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjob_title ON bucket0(job_title) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:43,474] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjob_title+ON+bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:43,686] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 206.994ms [2015-07-20 05:49:43,687] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjob_title ON default(job_title) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:43,698] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjob_title+ON+default%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:43,903] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 200.0066ms [2015-07-20 05:49:43,904] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employee008bd494b48243428ea41f81559bdf1cjob_title ON standard_bucket0(job_title) USING GSI WITH {'defer_build': True} [2015-07-20 05:49:43,917] - [rest_client:2237] INFO - query params : statement=CREATE+INDEX+employee008bd494b48243428ea41f81559bdf1cjob_title+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D [2015-07-20 05:49:44,110] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 177.9866ms [2015-07-20 05:49:46,283] - [base_2i:95] INFO - BUILD INDEX on bucket0(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:49:47,114] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on bucket0(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:49:47,127] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+bucket0%28employee008bd494b48243428ea41f81559bdf1cjoin_yr%2Cemployee008bd494b48243428ea41f81559bdf1cjob_title%29+USING+GSI [2015-07-20 05:49:47,419] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 287.0009ms [2015-07-20 05:49:47,420] - [base_2i:95] INFO - BUILD INDEX on default(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:49:48,421] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on default(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:49:48,438] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+default%28employee008bd494b48243428ea41f81559bdf1cjoin_yr%2Cemployee008bd494b48243428ea41f81559bdf1cjob_title%29+USING+GSI [2015-07-20 05:50:08,895] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 20.4510337s [2015-07-20 05:50:08,896] - [base_2i:95] INFO - BUILD INDEX on standard_bucket0(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:50:09,897] - [tuq_helper:72] INFO - RUN QUERY BUILD INDEX on standard_bucket0(employee008bd494b48243428ea41f81559bdf1cjoin_yr,employee008bd494b48243428ea41f81559bdf1cjob_title) USING GSI [2015-07-20 05:50:09,912] - [rest_client:2237] INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28employee008bd494b48243428ea41f81559bdf1cjoin_yr%2Cemployee008bd494b48243428ea41f81559bdf1cjob_title%29+USING+GSI [2015-07-20 05:50:30,256] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 20.3380026s [2015-07-20 05:50:30,271] - [basetestcase:309] INFO - sleep for 1 secs. ... [2015-07-20 05:50:31,258] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:31,272] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:31,272] - [base_2i:729] INFO - {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': True}, 'before': {'query_explain_ops': False, 'create_index': True, 'drop_index': False, 'query_ops': False}} [2015-07-20 05:50:31,582] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 301.0403ms [2015-07-20 05:50:31,582] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:31,593] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:31,947] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 346.9918ms [2015-07-20 05:50:41,958] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:41,972] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:42,446] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 462.001ms [2015-07-20 05:50:42,447] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:42,463] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:42,857] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 384.9997ms [2015-07-20 05:50:52,867] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:52,880] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:53,248] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 359.9731ms [2015-07-20 05:50:53,248] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:53,260] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:53,572] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 305.9808ms [2015-07-20 05:50:53,573] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:53,584] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:53,919] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 329.0312ms [2015-07-20 05:50:53,920] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:50:53,930] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:50:54,270] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 332.9932ms [2015-07-20 05:50:54,363] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.191&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.144%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91 [2015-07-20 05:50:54,376] - [rest_client:1170] INFO - rebalance operation started [2015-07-20 05:50:54,469] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-20 05:51:04,510] - [rest_client:1288] INFO - rebalance percentage : 1.56 % [2015-07-20 05:51:14,613] - [rest_client:1288] INFO - rebalance percentage : 3.31 % [2015-07-20 05:51:24,632] - [rest_client:1288] INFO - rebalance percentage : 4.18 % [2015-07-20 05:51:34,650] - [rest_client:1288] INFO - rebalance percentage : 4.18 % [2015-07-20 05:51:44,670] - [rest_client:1288] INFO - rebalance percentage : 5.00 % [2015-07-20 05:51:54,749] - [rest_client:1288] INFO - rebalance percentage : 6.96 % [2015-07-20 05:52:04,801] - [rest_client:1288] INFO - rebalance percentage : 8.53 % [2015-07-20 05:52:14,821] - [rest_client:1288] INFO - rebalance percentage : 9.96 % [2015-07-20 05:52:24,840] - [rest_client:1288] INFO - rebalance percentage : 9.96 % [2015-07-20 05:52:34,860] - [rest_client:1288] INFO - rebalance percentage : 9.96 % [2015-07-20 05:52:44,878] - [rest_client:1288] INFO - rebalance percentage : 9.96 % [2015-07-20 05:52:54,898] - [rest_client:1288] INFO - rebalance percentage : 9.96 % [2015-07-20 05:53:04,940] - [rest_client:1271] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed [2015-07-20 05:53:04,974] - [rest_client:2195] INFO - Latest logs from UI on 172.23.107.144: [2015-07-20 05:53:04,974] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.27919.31>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.107.145',\n {'EXIT',\n {{error,{badrpc,nodedown}},\n {gen_server,call,\n [{'janitor_agent-standard_bucket0',\n 'ns_1@172.23.107.145'},\n {if_rebalance,<0.19056.31>,\n {update_vbucket_state,362,replica,\n undefined,'ns_1@172.23.107.144'}},\n infinity]}}}}]}}}\n", u'shortText': u'message', u'serverTime': u'2015-07-20T05:52:49.809Z', u'module': u'ns_orchestrator', u'tstamp': 1437396769809, u'type': u'info'} [2015-07-20 05:53:04,975] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 0, u'text': u"<0.27865.31> exited with {unexpected_exit,\n {'EXIT',<0.27919.31>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.107.145',\n {'EXIT',\n {{error,{badrpc,nodedown}},\n {gen_server,call,\n [{'janitor_agent-standard_bucket0',\n 'ns_1@172.23.107.145'},\n {if_rebalance,<0.19056.31>,\n {update_vbucket_state,362,replica,\n undefined,'ns_1@172.23.107.144'}},\n infinity]}}}}]}}}", u'shortText': u'message', u'serverTime': u'2015-07-20T05:52:49.809Z', u'module': u'ns_vbucket_mover', u'tstamp': 1437396769809, u'type': u'critical'} [2015-07-20 05:53:04,975] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 0, u'text': u'Bucket "standard_bucket0" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2015-07-20T05:50:41.278Z', u'module': u'ns_vbucket_mover', u'tstamp': 1437396641278, u'type': u'info'} [2015-07-20 05:53:04,975] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 0, u'text': u'Started rebalancing bucket standard_bucket0', u'shortText': u'message', u'serverTime': u'2015-07-20T05:50:40.871Z', u'module': u'ns_rebalancer', u'tstamp': 1437396640871, u'type': u'info'} [2015-07-20 05:53:04,975] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.40','ns_1@172.23.107.144',\n 'ns_1@172.23.107.145','ns_1@172.23.105.91'], EjectNodes = ['ns_1@172.23.105.191'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2015-07-20T05:50:40.590Z', u'module': u'ns_orchestrator', u'tstamp': 1437396640590, u'type': u'info'} [2015-07-20 05:53:04,976] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.105.191', u'code': 0, u'text': u'Bucket "standard_bucket0" loaded on node \'ns_1@172.23.105.191\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-07-20T05:25:26.683Z', u'module': u'ns_memcached', u'tstamp': 1437395126683, u'type': u'info'} [2015-07-20 05:53:04,976] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.107.144\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2015-07-20 05:25:32.944477 : test_rebalance_out started \n', u'shortText': u'client-side error report', u'serverTime': u'2015-07-20T05:25:19.262Z', u'module': u'menelaus_web', u'tstamp': 1437395119262, u'type': u'warning'} [2015-07-20 05:53:04,976] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.105.191', u'code': 0, u'text': u'Bucket "bucket0" loaded on node \'ns_1@172.23.105.191\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-07-20T05:25:18.245Z', u'module': u'ns_memcached', u'tstamp': 1437395118245, u'type': u'info'} [2015-07-20 05:53:04,977] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.105.191', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.191\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-07-20T05:25:12.870Z', u'module': u'ns_memcached', u'tstamp': 1437395112870, u'type': u'info'} [2015-07-20 05:53:04,977] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.145', u'code': 0, u'text': u'Bucket "standard_bucket0" loaded on node \'ns_1@172.23.107.145\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2015-07-20T05:25:11.364Z', u'module': u'ns_memcached', u'tstamp': 1437395111364, u'type': u'info'} ERROR [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', ''s__beolf.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 73, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 445, 'check', 'self.set_exception(ex)otstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2'.7/u)ni, tte(st/ca'slib/etasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] .py'Mon Jul 20 05:53:04 2015 , 369, 'run', 'testMethod()'), ('pytests/2i/recovery_2i.py', 61, 'test_rebalance_out', 'rebalance.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] [2015-07-20 05:53:04,979] - [recovery_2i:29] INFO - <<<<<< WILL DROP THE INDEXES >>>>> [2015-07-20 05:53:05,048] - [basetestcase:1697] INFO - list of all nodes in cluster: [ip:172.23.107.144 port:8091 ssh_username:Administrator] [2015-07-20 05:53:05,049] - [tuq_helper:382] INFO - CHECK FOR PRIMARY INDEXES [2015-07-20 05:53:05,049] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON bucket0 USING VIEW [2015-07-20 05:53:05,049] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:53:05,102] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:53:05,811] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 701.001ms [2015-07-20 05:53:05,812] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON bucket0 USING VIEW [2015-07-20 05:53:05,882] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+bucket0+USING+VIEW+ [2015-07-20 05:53:05,896] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 9.0002ms [2015-07-20 05:53:05,896] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON default USING VIEW [2015-07-20 05:53:05,897] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:53:05,910] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:53:07,412] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.4949788s [2015-07-20 05:53:07,413] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING VIEW [2015-07-20 05:53:07,453] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+VIEW+ [2015-07-20 05:53:07,470] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 8.9909ms [2015-07-20 05:53:07,470] - [tuq_helper:389] INFO - DROP PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-20 05:53:07,470] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes [2015-07-20 05:53:07,483] - [rest_client:2237] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes [2015-07-20 05:53:09,213] - [tuq_helper:94] INFO - TOTAL ELAPSED TIME: 1.7199974s [2015-07-20 05:53:09,213] - [tuq_helper:72] INFO - RUN QUERY DROP PRIMARY INDEX ON standard_bucket0 USING VIEW [2015-07-20 05:53:09,233] - [rest_client:2237] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+standard_bucket0+USING+VIEW+ [2015-07-20 05:53:10,434] - [tuq_helper:395] ERROR - ERROR during index creation CBQError: host 172.23.107.144: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'Failed to drop index #primary - cause: dial tcp 172.23.107.145:8092: ConnectEx tcp: No connection could be made because the target machine actively refused it.', u'code': 13006}], u'results': [], u'metrics': {u'elapsedTime': u'1.1759985s', u'executionTime': u'1.1759985s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'cde068d9-a4b9-4f93-bd0e-0041a5db4c14', u'signature': None} [2015-07-20 05:53:11,158] - [remote_util:754] INFO - File /cygdrive/c/Windows/win.ini was found [2015-07-20 05:53:11,158] - [remote_util:2108] INFO - This is windows server! [2015-07-20 05:53:11,255] - [remote_util:754] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2015-07-20 05:53:11,355] - [remote_util:795] INFO - get windows information [2015-07-20 05:53:11,366] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: hostname [2015-07-20 05:53:11,534] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:11,535] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: echo "ipconfig" > /tmp/cmd.bat; /tmp/cmd.bat [2015-07-20 05:53:11,811] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:11,812] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: rm -f /opt/couchbase/var/lib/couchbase/meta;rm -f /tmp/log_upr_client.sock [2015-07-20 05:53:12,180] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:12,181] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: killall cbq-engine [2015-07-20 05:53:12,438] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:12,439] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: killall tuqtng [2015-07-20 05:53:12,619] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:12,620] - [remote_util:1919] INFO - running command.raw on 172.23.107.144: killall indexer [2015-07-20 05:53:12,819] - [remote_util:1956] INFO - command executed successfully [2015-07-20 05:53:12,820] - [basetestcase:241] INFO - ============== basetestcase cleanup was started for test #7 test_rebalance_out ============== [2015-07-20 05:53:12,971] - [bucket_helper:137] INFO - deleting existing buckets [u'bucket0', u'default', u'standard_bucket0'] on 172.23.107.144 [2015-07-20 05:53:12,972] - [bucket_helper:139] INFO - remove bucket bucket0 ... [2015-07-20 05:53:19,114] - [bucket_helper:153] INFO - deleted bucket : bucket0 from 172.23.107.144 [2015-07-20 05:53:19,115] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-20 05:53:19,322] - [rest_client:124] INFO - node 172.23.107.144 existing buckets : [u'default', u'standard_bucket0'] [2015-07-20 05:53:19,323] - [bucket_helper:139] INFO - remove bucket default ... [2015-07-20 05:53:21,624] - [bucket_helper:153] INFO - deleted bucket : default from 172.23.107.144 [2015-07-20 05:53:21,625] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-20 05:53:21,673] - [rest_client:124] INFO - node 172.23.107.144 existing buckets : [u'standard_bucket0'] [2015-07-20 05:53:21,675] - [bucket_helper:139] INFO - remove bucket standard_bucket0 ... [2015-07-20 05:53:24,392] - [bucket_helper:153] INFO - deleted bucket : standard_bucket0 from 172.23.107.144 [2015-07-20 05:53:24,392] - [bucket_helper:226] INFO - waiting for bucket deletion to complete.... [2015-07-20 05:53:24,457] - [rest_client:124] INFO - node 172.23.107.144 existing buckets : [] [2015-07-20 05:53:24,474] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.145 [2015-07-20 05:53:24,501] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.191 [2015-07-20 05:53:24,519] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.40 [2015-07-20 05:53:24,535] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.91 [2015-07-20 05:53:24,548] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.88 [2015-07-20 05:53:24,626] - [cluster_helper:260] INFO - rebalancing all nodes in order to remove nodes [2015-07-20 05:53:24,632] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.144%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91 [2015-07-20 05:53:24,647] - [rest_client:1170] INFO - rebalance operation started [2015-07-20 05:53:24,696] - [rest_client:1288] INFO - rebalance percentage : 0.00 % [2015-07-20 05:53:34,720] - [rest_client:1243] INFO - rebalance progress took 10.0725710392 seconds [2015-07-20 05:53:34,720] - [rest_client:1244] INFO - sleep for 10 seconds after rebalance... [2015-07-20 05:53:44,786] - [cluster_helper:300] INFO - removed all the nodes from cluster associated with ip:172.23.107.144 port:8091 ssh_username:Administrator ? [(u'ns_1@172.23.105.191', 8091), (u'ns_1@172.23.105.40', 8091), (u'ns_1@172.23.107.145', 8091), (u'ns_1@172.23.105.91', 8091)] [2015-07-20 05:53:44,787] - [basetestcase:309] INFO - sleep for 10 secs. ... [2015-07-20 05:53:54,807] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.144:8091 [2015-07-20 05:53:54,815] - [cluster_helper:80] INFO - ns_server @ 172.23.107.144:8091 is running [2015-07-20 05:53:54,821] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.145:8091 [2015-07-20 05:53:54,828] - [cluster_helper:80] INFO - ns_server @ 172.23.107.145:8091 is running [2015-07-20 05:53:54,835] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.191:8091 [2015-07-20 05:53:54,842] - [cluster_helper:80] INFO - ns_server @ 172.23.105.191:8091 is running [2015-07-20 05:53:54,848] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.40:8091 [2015-07-20 05:53:54,854] - [cluster_helper:80] INFO - ns_server @ 172.23.105.40:8091 is running [2015-07-20 05:53:54,861] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.91:8091 [2015-07-20 05:53:54,868] - [cluster_helper:80] INFO - ns_server @ 172.23.105.91:8091 is running [2015-07-20 05:53:54,875] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.88:8091 [2015-07-20 05:53:54,882] - [cluster_helper:80] INFO - ns_server @ 172.23.105.88:8091 is running [2015-07-20 05:53:54,882] - [basetestcase:260] INFO - ============== basetestcase cleanup was finished for test #7 test_rebalance_out ============== ====================================================================== ERROR: test_rebalance_out (2i.recovery_2i.SecondaryIndexingRecoveryTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/2i/recovery_2i.py", line 61, in test_rebalance_out rebalance.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 1839.862s