Description
Build
5.0.0-3358
Mihir spotted this error while running a 2i test with Sarath on the longevity cluster. From the error it appears to come from a n invalid json parse error. Also from the cluster, key 00000000000000000002 contains binary data.
At this point this all the info I have -
{code}
[10:08 AM] Mihir Kamdar: I was helping Sarath to reproduce a 2i issue on the Centos system testing cluster, and I used the longevity test to setup the cluster, but then aborted after the data that I wanted got created. I saw an FTS crash in the logs so thought of letting you know if you want to investigate
[10:08 AM] Mihir Kamdar: Service 'fts' exited with status 1. Restarting. Messages:
2017-07-25T09:12:57.110-07:00 [INFO] bleve: json.Unmarshal, partition: 651, key: "00000000000000000002", seq: 152085, err: invalid character '#' looking for beginning of value
2017-07-25T09:12:57.110-07:00 [INFO] bleve: json.Unmarshal, partition: 591, key: "00000000000000004105", seq: 101776, err: invalid character '#' looking for beginning of value
2017-07-25T09:12:57.110-07:00 [INFO] main: ------------------------------------------------------
2017-07-25T09:12:57.110-07:00 [INFO] init_http: web UI / REST API is available: https://:18094
2017-07-25T09:12:57.110-07:00 [INFO] main: ------------------------------------------------------
2017-07-25T09:12:57.111-07:00 [INFO] bleve: json.Unmarshal, partition: 153, key: "00000000000000000412", seq: 114444, err: invalid character '#' looking for beginning of value
2017-07-25T09:12:57.111-07:00 [FATA] init_http: listen, err: listen tcp :18094: bind: address already in use – main.mainServeHTTP() at init_http.go:135
[goport(/opt/couchbase/bin/cbft)] 2017/07/25 09:13:02 child process exited with status 1
[10:08 AM] Mihir Kamdar: http://172.23.108.103:8091 is the master node
[10:09 AM] Mihir Kamdar: pls let the system be in the same state since we are still working on the issue, but you can check this FTS issue
[10:09 AM] Mihir Kamdar: thanks
{code}
Logs attached - .188 and .242 are full text search nodes.
The restart is on .188. https://s3.amazonaws.com/cb-engineering/Aruna/collectinfo-2017-07-25T180619-ns_1%40172.23.106.188.zip
The log is filled with json Unmarshall errors that the log doesn't contain the context of the error anymore.
Cbcollect from other nodes are also available.