Details
-
Bug
-
Resolution: Cannot Reproduce
-
Blocker
-
4.0.0
-
Security Level: Public
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Build
------
4.0.0-3255
Found during all-component system test.
Steps
------
- 8 nodes in cluster, 2 with query services, one with index, all nodes have kv.
- created 9 tpcc buckets
- start loading on all buckets, create index in parallel
- building index on STOCK bucket was stuck at 22%. Interestingly, it went down to 18% and then to 20% and remained so. The next index creation stmt (on WAREHOUSE) timed out.
- .50 is the index node. Attaching cbcollect from the same.
Live-cluster : http://172.23.105.44:8091/index.html
Also noticed very high memory and cpu usage > 95% on .50. This could also be a sizing issue, dev to confirm. I had to stop Couchbase to even run cbcollect. After restarting, indexer was heavily using cpu but index remained stuck. Stats show 4.15M keys yet to be indexed.
|
|
> create index ST_W_ID_I_ID1 on STOCK(S_W_ID, S_I_ID) using gsi WITH {"defer_build":true}
|
{u'status': u'success', u'metrics': {u'elapsedTime': u'56.475985046s', u'executionTime': u'56.475783132s', u'resultSize': 0, u'resultCount': 0}, u'results': [], u'requestID': u'ec861526-f9d8-4f0d-9088-ddcaa2da843e', u'signature': None}
|
|
|
> build index on STOCK(ST_W_ID_I_ID1) using gsi
|
{u'status': u'success', u'metrics': {u'elapsedTime': u'4.290571608s', u'executionTime': u'4.290451479s', u'resultSize': 0, u'resultCount': 0}, u'results': [], u'requestID': u'36f1991d-0dd6-463b-940c-4d66efdba995', u'signature': None}
|
|
|
> create index WH_ID on WAREHOUSE(W_ID) using gsi WITH {"defer_build":true}
|
{u'status': u'errors', u'errors': [{u'msg': u'GSI CreateIndex() - cause: Request Timeout', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'2m0.477918484s', u'executionTime': u'2m0.477809858s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'7f02eb62-fa35-4ca2-b888-7537103506f2', u'signature': None}
|
|
|
> build index on WAREHOUSE(WH_ID) using gsi
|
{u'status': u'errors', u'errors': [{u'msg': u'GSI index WH_ID not found.', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'3.969093ms', u'executionTime': u'3.85548ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'0f6df765-08ae-4a80-818d-9b3867b9955e', u'signature': None}
|
|
|
>select keyspace_id, state from system:indexes where state != 'online'
|
{u'status': u'success', u'metrics': {u'elapsedTime': u'3.343423427s', u'executionTime': u'3.34328641s', u'resultSize': 78, u'resultCount': 1}, u'results': [{u'keyspace_id': u'STOCK', u'state': u'pending'}], u'requestID': u'0474cfd9-3f5e-4606-a4d0-839943c4512e', u'signature': {u'keyspace_id': u'json', u'state': u'json'}}
|
|
Screenshot attached.
