cbq-engine and indexer are getting killed

Test which fails:

./testrunner -i /tmp/n1ql_set3.ini gsi_type=memory_optimized,nodes_init=1,get-cbcollect-info=True -t tuqquery.tuq_tokens.TokenTests.test_tokens_secondary_indexes

This test loads beer-sample, creates primary index on the bucket and then creates 18 secondary indexes :

create index idx1 on `beer-sample`(description,name )

create index idx2 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description) END ,description,name )

create index idx3 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"lower","names":true,"specials":false}) END ,description,name )

create index idx4 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper","names":false,"specials":true}) END ,description,name )

create index idx5 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper","names":false}) END ,description,name )

create index idx6 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper"}) END ,description,name )

create index idx7 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{}) END ,description,name )

 create index idx8 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"":""}) END ,description,name )

 create index idx9 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"random"}) END ,description,name )

 create index idx10 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"names":"random"}) END ,description,name )

create index idx11 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"specials":"random"}) END ,description,name )

create index idx12 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description) END )

 create index idx13 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"lower"}) END  )

create index idx14 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper"}) END )

create index idx15 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"lower","names":true,"specials":false}) END  )

create index idx16 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper","names":false,"specials":true}) END  )

create index idx17 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{"case":"upper","names":false}) END )

create index idx18 on `beer-sample`( DISTINCT ARRAY v FOR v in tokens(description,{}) END )

Error reported:

2017-02-03 13:58:56 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://172.23.107.94:8093/query?scan_consistency=REQUEST_PLUS&creds=%5B%5D&statement=create+index+idx18+on+%60beer-sample%60%28+DISTINCT+ARRAY+v+FOR+v+in+tokens%28description%2C%7B%7D%29+END+%29 error timed out 

ERROR

2017-02-03 13:58:56 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.94 with username:root password:couchbase ssh_key:

2017-02-03 13:58:56 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.94

2017-02-03 13:58:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs.  ...

======================================================================

ERROR: test_tokens_secondary_indexes (tuqquery.tuq_tokens.TokenTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "pytests/tuqquery/tuq_tokens.py", line 83, in test_tokens_secondary_indexes

    self.run_cbq_query()

  File "pytests/tuqquery/tuq_tokens.py", line 254, in run_cbq_query

    result = RestConnection(server).query_tool(query, self.n1ql_port, query_params=query_params)

  File "./lib/membase/api/rest_client.py", line 2776, in query_tool

    status, content, header = self._http_request(api, 'POST', timeout=timeout, headers=headers)

  File "./lib/membase/api/rest_client.py", line 795, in _http_request

    raise ServerUnavailableException(ip=self.ip)

ServerUnavailableException: unable to reach the host @ 172.23.107.94

----------------------------------------------------------------------

Ran 1 test in 777.834s

Attaching Log.

I see these errors in indexer.log file:

2017-02-03T05:46:35.308-08:00 [Info] timekeeper::generateNewStability: setting snapshot type as DISK_SNAP due to BuildCompletionTS

2017-02-03T05:46:35.309-08:00 [Error] Encoded array key too long. Length of key = 31613, Limit = 30978

2017-02-03T05:46:35.309-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: brouwerij_bosteels-pauwel_kwak in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.309-08:00 [Error] Encoded array key too long. Length of key = 31793, Limit = 30978

2017-02-03T05:46:35.309-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: boston_beer_company-samuel_adams_hallertau_imperial_pilsner in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.309-08:00 [Error] Encoded array key too long. Length of key = 31465, Limit = 30978

2017-02-03T05:46:35.309-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: brouwerij_van_steenberge-piraat_10_5 in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.309-08:00 [Error] Encoded array key too long. Length of key = 31649, Limit = 30978

2017-02-03T05:46:35.309-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: barley_creek_brewing in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.310-08:00 [Error] Encoded array key too long. Length of key = 31306, Limit = 30978

2017-02-03T05:46:35.310-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: rogue_ales-morimoto_imperial_pilsner in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.310-08:00 [Error] Encoded array key too long. Length of key = 31001, Limit = 30978

2017-02-03T05:46:35.311-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: bear_republic_brewery-racer_5_ipa in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

2017-02-03T05:46:35.311-08:00 [Error] Encoded array key too long. Length of key = 31254, Limit = 30978

2017-02-03T05:46:35.311-08:00 [Error] MemDBSlice::insertSecArrayIndex Error indexing docid: erie_brewing_company-sunshie_wit in Slice: 0. Error: Encoded array key too long (> 30978). Skipped.

This is a regression from 5.0.0-1873 build.

Job which is broken:

http://qa.sc.couchbase.com/job/cen006-p0-n1ql-vset03-07-covering_gsi/