CC is dead and cluster is in UNUSABLE state when running system test with secondary indexes with 2 days.

• 4 nodes in cluster: 2 KV, 2CBAS
• 4 buckets in CB and 4 corresponding buckets in CBAS shadowing 4 datasets.
• Secondary indexes present on 2 datasets.

  create index idx_result1 on ds1(result:string);

  create index idx_result2 on ds2(result:string);

• 12 queries/sec running on all 4 datasets makes it 48 queries/sec.
• 2 query types are such that they use secondary indexes while other 2 use primary index.

  select COUNT(result) from ds1 where result = "SUCCESS" limit 50

  select claim from ds2 where result != "SUCCESS" limit 50

  select COUNT(result) from ds3 where result = "SUCCESS" limit 50

  select * from ds4 where result != "SUCCESS" limit 50

On CC:
Failed to get bucket insights.
Analytics service returned status: 500
Try refreshing the bucket insights.

On workbench(CC):

[

  {

    "code": 1,

    "msg": "Exception: Cannot execute request, cluster is UNUSABLE",

    "query_from_user": "select count(*) from ds3"

  }

]

On NC:
Failed to get bucket insights.
Analytics service returned status: 504
Try refreshing the bucket insights.