[GSI][Upgrade] Indexer add back fails while upgrading from 5.1.0 to 5.5.0

{u'node': u'ns_1@172.23.105.121', u'code': 0, u'text': u"Service 'indexer' exited with status 134. Restarting. Messages:\nruntime.goparkunlock(0xc423524b38, 0xf8a1bd, 0xc, 0xc423114717, 0x3)\n\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/proc.go:265 +0x5e fp=0xc4231146c8 sp=0xc423114688\nruntime.chanrecv(0xddbee0, 0xc423524ae0, 0x0, 0x1, 0x0)\n\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/chan.go:496 +0x2df fp=0xc423114750 sp=0xc4231146c8\nruntime.chanrecv1(0xddbee0, 0xc423524ae0, 0x0)\n\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/chan.go:378 +0x35 fp=0xc423114788 sp=0xc423114750\ngithub.com/couchbase/plasma.(*Plasma).swapperDaemon.func2(0xc4232c0a80, 0xc4234dc0c0)\n\tgoproj/src/github.com/couchbase/plasma/swapper.go:148 +0x47 fp=0xc4231147b0 sp=0xc423114788\nruntime.goexit()\n\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4231147b8 sp=0xc4231147b0\ncreated by github.com/couchbase/plasma.(*Plasma).swapperDaemon\n\tgoproj/src/github.com/couchbase/plasma/swapper.go:150 +0x16d\n[goport(/opt/couchbase/bin/indexer)] 2018/06/01 00:12:55 child process exited with status 134\n", u'shortText': u'message', u'serverTime': u'2018-06-01T00:12:56.032Z', u'module': u'ns_log', u'tstamp': 1527837176032, u'type': u'info'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.105.101', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,index,\n                                 {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2018-06-01T00:12:55.796Z', u'module': u'ns_orchestrator', u'tstamp': 1527837175796, u'type': u'critical'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.105.101', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-06-01T00:12:55.044Z', u'module': u'ns_vbucket_mover', u'tstamp': 1527837175044, u'type': u'info'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.105.101', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2018-06-01T00:12:54.867Z', u'module': u'ns_rebalancer', u'tstamp': 1527837174867, u'type': u'info'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.105.101', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.101','ns_1@172.23.105.121',\n                                 'ns_1@172.23.107.62'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2018-06-01T00:12:54.764Z', u'module': u'ns_orchestrator', u'tstamp': 1527837174764, u'type': u'info'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.105.121', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.121\'. Version: "5.5.0-2807-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2018-06-01T00:10:26.497Z', u'module': u'menelaus_sup', u'tstamp': 1527837026497, u'type': u'info'}

2018-06-01 00:13:04,879 - root - ERROR - {u'node': u'ns_1@172.23.107.62', u'code': 4, u'text': u"Node 'ns_1@172.23.107.62' saw that node 'ns_1@172.23.105.121' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2018-06-01T00:10:26.297Z', u'module': u'ns_node_disco', u'tstamp': 1527837026297, u'type': u'info'}

2018-06-01 00:13:04,880 - root - ERROR - {u'node': u'ns_1@172.23.105.101', u'code': 4, u'text': u"Node 'ns_1@172.23.105.101' saw that node 'ns_1@172.23.105.121' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2018-06-01T00:10:26.241Z', u'module': u'ns_node_disco', u'tstamp': 1527837026241, u'type': u'info'}

2018-06-01 00:13:04,880 - root - ERROR - {u'node': u'ns_1@172.23.105.121', u'code': 2, u'text': u'Node \'ns_1@172.23.105.121\' synchronized otp cookie {sanitized,\n                                                    <<"DKqKDOAd14heCiUkzpXvEG+BZzIiWLJMGiisnrSErsk=">>} from cluster', u'shortText': u'cookie update', u'serverTime': u'2018-06-01T00:10:26.232Z', u'module': u'ns_cookie_manager', u'tstamp': 1527837026232, u'type': u'info'}

2018-06-01 00:13:04,880 - root - ERROR - {u'node': u'ns_1@172.23.107.62', u'code': 5, u'text': u"Node 'ns_1@172.23.107.62' saw that node 'ns_1@172.23.105.121' went down. Details: [{nodedown_reason,\n                                                                                    connection_closed}]", u'shortText': u'node down', u'serverTime': u'2018-06-01T00:09:04.098Z', u'module': u'ns_node_disco', u'tstamp': 1527836944098, u'type': u'warning'}

./testrunner -i /tmp/centos_upgrade2i_4_5_1_green.ini -p upgrade_to=5.5.0-2807,initial_version=5.1.0-5552,get-cbcollect-info=True,get-coredumps=True -t 2i.upgrade_2i.UpgradeSecondaryIndex.test_online_upgrade_with_failover,nodes_init=3,services_init=kv:n1ql-index-index,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,gsi_type=forestdb,nodes_out=2,nodes_out_dist=index:2,nodes_in=2,services_in=index:2,before=create_index,disable_plasma_upgrade=False,build_index_after_create=False