[System Test] Panic after losing checkpoint blobs from metadata bucket after KV hard failover recovery

[2018-06-26T04:04:04-07:00, sequoiatools/couchbase-cli:475400] server-add -c 172.23.108.103:8091 --server-add 172.23.108.104:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data

[2018-06-26T04:08:32-07:00, sequoiatools/couchbase-cli:a4c4b2] failover -c 172.23.108.103:8091 --server-failover 172.23.98.135:8091 -u Administrator -p password

[2018-06-26T04:23:17-07:00, sequoiatools/couchbase-cli:8eea83] failover -c 172.23.108.103:8091 --server-failover 172.23.96.148:8091 -u Administrator -p password --force

[2018-06-26T04:23:48-07:00, sequoiatools/couchbase-cli:92cb4f] rebalance -c 172.23.108.103:8091 -u Administrator -p password

Service 'eventing' exited with status 134. Restarting. Messages: godeps/src/gopkg.in/couchbase/gocbcore.v7/agentrouting.go:165 +0x25b fp=0xc427d9fec0 sp=0xc427d9fe28 gopkg.in/couchbase/gocbcore%2ev7.(*Agent).(gopkg.in/couchbase/gocbcore%2ev7.slowDialMemdClient)-fm(0xc42050f420, 0x14, 0xc421d06f50, 0x1, 0x1) godeps/src/gopkg.in/couchbase/gocbcore.v7/agentrouting.go:224 +0x3e fp=0xc427d9ff00 sp=0xc427d9fec0 gopkg.in/couchbase/gocbcore%2ev7.newMemdClientMux.func1(0xdc0a78, 0x35, 0xc421d06f50) godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclientmux.go:22 +0x3a fp=0xc427d9ff38 sp=0xc427d9ff00 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run(0xc4265ccd80) godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:148 +0x1d8 fp=0xc427d9ffa8 sp=0xc427d9ff38 runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc427d9ffb0 sp=0xc427d9ffa8 created by gopkg.in/couchbase/gocbcore%2ev7.(*memdPipeline).StartClients godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipeline.go:70 +0x127 [goport(/opt/couchbase/bin/eventing-producer)] 2018/06/26 04:40:16 child process exited with status 134

Rebalance exited with reason {service_rebalance_failed,eventing, {badmatch, {error, {unknown_error, <<"Some apps are undergoing bootstrap">>}}}}

Service 'projector' exited with status 134. Restarting. Messages: 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/syscall/syscall_unix.go:161 +0x49 fp=0xc420030e90 sp=0xc420030e48 os.(*File).read(0xc420032008, 0xc420030f7c, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/os/file_unix.go:228 +0x4f fp=0xc420030ed8 sp=0xc420030e90 os.(*File).Read(0xc420032008, 0xc420030f7c, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/os/file.go:101 +0x59 fp=0xc420030f40 sp=0xc420030ed8 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() goproj/src/github.com/couchbase/indexing/secondary/common/util.go:355 +0x55 fp=0xc420030fc0 sp=0xc420030f40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420030fc8 sp=0xc420030fc0 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/projector/main.go:110 +0x6f3 [goport(/opt/couchbase/bin/projector)] 2018/06/26 05:35:32 child process exited with status 134