Build Sanity (7.0.0-4902) - Regression test cases are failing

Indexing service unavailable.

[2021-04-08 23:54:38,880] - [rest_client:1018] ERROR - POST
http://172.23.106.48:8093/query?statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI
body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '/'} error: 500 reason: unknown b'{\n"requestID": "bf162960-6974-4850-b1b6-cf36f1c6e92e",\n"signature": null,\n"results": [\n],\n"errors": [

{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index creation for index #primary, bucket dst_bucket, scope _default, collection _default cannot start. Reason: Fails to create index. There is no available index service that can process this request at this time. Index Service can be in bootstrap, recovery, or non-reachable. Please retry the operation at a later time.."}

],\n"status": "errors",\n"metrics": {"elapsedTime": "25.009316454s","executionTime": "25.009242967s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password
[2021-04-08 23:54:38,881] - [tuq_helper:710] ERROR - ERROR during index creation CBQError: host 172.23.106.48: ERROR:{'requestID': 'bf162960-6974-4850-b1b6-cf36f1c6e92e', 'signature': None, 'results': [], 'errors': [

{'code': 5000, 'msg': 'GSI CreatePrimaryIndex() - cause: Index creation for index #primary, bucket dst_bucket, scope _default, collection _default cannot start. Reason: Fails to create index. There is no available index service that can process this request at this time. Index Service can be in bootstrap, recovery, or non-reachable. Please retry the operation at a later time..'}

], 'status': 'error
 
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.26', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1617949176711, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.11384.0>,\n {rebalance_failed,\n

{service_error,\n <<"failed to aggregate rebalance progress from all eventing nodes, err: map[127.0.0.1:8096:Get   +*Rebalance failure:*+   [http://127.0.0.1:8096/getAggRebalanceProgress] : net/http: request canceled (Client.Timeout exceeded while awaiting headers)]">>}

}}}}.\nRebalance Operation Id = 97fd30e00caafb95413ef0cd20318104', 'serverTime': '2021-04-08T23:19:36.711Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1617949099190, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-04-08T23:18:19.190Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.26', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1617949099103, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.105.26','ns_1@172.23.105.33',\n 'ns_1@172.23.105.34','ns_1@172.23.105.35',\n 'ns_1@172.23.105.44','ns_1@172.23.105.45'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 97fd30e00caafb95413ef0cd20318104", 'serverTime': '2021-04-08T23:18:19.103Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1617949099078, 'shortText': 'message', 'text': 'Node ns_1@172.23.105.44 joined cluster', 'serverTime': '2021-04-08T23:18:19.078Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1617949099035, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.44\'. Version: "7.0.0-12642-enterprise".', 'serverTime': '2021-04-08T23:18:19.035Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.33', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1617949097123, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.105.33' saw that node 'ns_1@172.23.105.44' came up. Tags: []", 'serverTime': '2021-04-08T23:18:17.123Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.34', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1617949097123, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.105.34' saw that node 'ns_1@172.23.105.44' came up. Tags: []", 'serverTime': '2021-04-08T23:18:17.123Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.45', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1617949097121, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.105.45' saw that node 'ns_1@172.23.105.44' came up. Tags: []", 'serverTime': '2021-04-08T23:18:17.121Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.35', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1617949097121, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.105.35' saw that node 'ns_1@172.23.105.44' came up. Tags: []", 'serverTime': '2021-04-08T23:18:17.121Z'}
[2021-04-08 23:19:39,313] - [rest_client:3769] ERROR - {'node': 'ns_1@172.23.105.26', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1617949096909, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.105.26' saw that node 'ns_1@172.23.105.44' came up. Tags: []", 'serverTime': '2021-04-08T23:18:16.909Z'}