Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
4.0.0
-
Security Level: Public
-
Windows
-
Untriaged
-
Yes
Description
Run - http://qa.sc.couchbase.com/job/w012R2-2i-vset03-03-recoverytests-add-index/19/consoleFull
Test case: test_1
./testrunner -i /tmp/recoverytests-add-index.ini doc-per-day=50,standard_buckets=1,sasl_buckets=1,skip_build_tuq=True,cbq_version=sherlock,scan_consistency=request_plus -t 2i.recovery_2i.SecondaryIndexingRecoveryTests.test_rebalance_in,initial=,before=create_index,after=query_ops,groups=simple,dataset=default,doc-per-day=10,nodes_init=5,nodes_in=1,services_in=kv,services_init=n1ql:kv-kv-kv-index-index,GROUP=REB-IN;P1
The build of index employee40d14b2f41f744a5b85637dc7724f198join_yr does not complete in 10 mins after which the test fails.
Test steps:
1. Create 3 buckets, load 100800 documents
2. Create a bunch of indexes (and build them fully)
3. Create a few more indexes including employee40d14b2f41f744a5b85637dc7724f198join_yr and start build of the indexes
4. While indexes are building, perform rebalance in of KV node:
adding remote node @172.23.105.88:8091 to this cluster @172.23.105.191:8091
Build index of employee40d14b2f41f744a5b85637dc7724f198join_yr is hung. Queries executed and their timestamps are:
2015-07-16 19:18:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON bucket0(join_yr) USING GSI WITH
{'defer_build': True}2015-07-16 19:18:17 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+bucket0%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 353.0029ms
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON default(join_yr) USING GSI WITH
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+default%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 213.0406ms
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee40d14b2f41f744a5b85637dc7724f198join_yr ON standard_bucket0(join_yr) USING GSI WITH
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee40d14b2f41f744a5b85637dc7724f198join_yr+ON+standard_bucket0%28join_yr%29+USING+GSI+WITH%7B%27defer_build%27%3A+True%7D
2015-07-16 19:18:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 307.9907ms
2015-07-16 19:18:20 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:20 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+bucket0%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
2015-07-16 19:18:21 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 475.0493ms
2015-07-16 19:18:21 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on default(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:22 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
2015-07-16 19:18:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.583002s
2015-07-16 19:18:43 | INFO | MainProcess | test_thread | [base_2i.async_build_index] BUILD INDEX on standard_bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on standard_bucket0(employee40d14b2f41f744a5b85637dc7724f198job_title,employee40d14b2f41f744a5b85637dc7724f198join_yr) USING GSI
2015-07-16 19:18:44 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+standard_bucket0%28employee40d14b2f41f744a5b85637dc7724f198job_title%2Cemployee40d14b2f41f744a5b85637dc7724f198join_yr%29+USING+GSI
2015-07-16 19:19:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.0490009s
Logs:
https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.191.zip
https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.240.zip
https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.105.91.zip
https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.107.144.zip
https://s3-ap-southeast-1.amazonaws.com/indexing-test/addIndex_Windows_19/collectinfo-2015-07-17T085846-ns_1%40172.23.107.145.zip
Cluster config:
[SERVER_1]
ip:172.23.105.191
services=n1ql,kv
[SERVER_2]
ip:172.23.105.240
services=kv
[SERVER_3]
ip:172.23.107.144
services=kv
[SERVER_4]
ip:172.23.107.145
services=index
[SERVER_5]
ip:172.23.105.91
services=index
[SERVER_6]
ip:172.23.105.88
services=kv
Note: This test does not fail on Centos