bucket fails to initialize when do offline upgrade from 5.0.0 to 6.5.0-1316

Offline upgrade from 5.0.0 to Mad-Hatter failed due to indexer crashed frequently.

Create 3 nodes cluster with version 5.0.0 with services kv,index-kv,n1ql and kv,fts.

Create default bucket

Download build 6.5.0-1316 to 3 nodes and do offline upgrade in those 3 nodes to 6.5.0

After upgrade were done, cluster become unstable.

Check log page, I saw a lot of errrors about indexer and compaction crashed

 
Compactor for database `default` (pid [{type,database}, {important,true}, {name,<<"default">>}, {fa, {#Fun<compaction_daemon.4.121429348>, [<<"default">>, {config, {undefined,undefined}, {undefined,undefined}, undefined,false,false, {daemon_config,30,131072,20971520}}, false, {[\{type,bucket}]}]}}]) terminated unexpectedly: {{bad_return_value, {stop, ...show compaction_daemon 000 ns_1@172.23.98.73 3:06:05 PM   Thu Sep 13, 2018
Service 'indexer' exited with status 1. Restarting. Messages: 2018-09-13T15:06:00.649-07:00 [Warn] Indexer::getCurrentKVTs error=MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: Retrying (8) 2018-09-13T15:06:00.718-07:00 [Error] StartDcpFeedOver(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 2018-09-13T15:06:00.719-07:00 [Warn] Indexer::getCurrentKVTs error=MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: Retrying (9) 2018-09-13T15:06:00.791-07:00 [Error] StartDcpFeedOver(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 2018-09-13T15:06:00.792-07:00 [Warn] Indexer::getCurrentKVTs error=MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: Retrying (10) 2018-09-13T15:06:00.868-07:00 [Error] StartDcpFeedOver(): MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 2018-09-13T15:06:00.869-07:00 [Error] Indexer::getCurrentKVTs Error Connecting to KV Cluster MCResponse status=KEY_ENOENT, opcode=0x89, opaque=0, msg: 2018-09-13T15:06: ...show
I will check which version of Mad-Hatter passed this test.