[BP MB-45666] [System Test] Panic seen in indexer logs - "panic in Follower.startListener() : runtime error: invalid memory address or nil pointer dereference"

2021-04-13T22:37:16.797-07:00 [Info] Fetch new stats upon request by life cycle manager

2021-04-13T22:37:16.939-07:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.107.3:56442.  Error = EOF. Kill Pipe.

2021-04-13T22:37:16.939-07:00 [Info] messageListener.start(): message channel closed. Remove peer 22:8b:be:e0:dd:38:42:88 as follower.

2021-04-13T22:37:16.994-07:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CLIENT_STATS elapsed 26.838µs len(expediates) 0 len(incomings) 0 len(outgoings) 0

2021-04-13T22:37:17.448-07:00 [Info] bucket3/idx1_ddEniCa/Mainstore#17055069458182677909:3 Plasma: SMR reclaim pending is higher than expected: pending = 464 KB (expected = 217 KB), wCtxCnt = 7, objCnt 14, changed reclaimList flush threshold from 1 to 0, changed reclaimSize flush threshold from 24 KB to 24 KB.

2021-04-13T22:37:17.467-07:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.107.5:55988.  Error = EOF. Kill Pipe.

2021-04-13T22:37:17.467-07:00 [Info] messageListener.start(): message channel closed. Remove peer 2d:69:78:70:ed:17:b5:2e as follower.

2021-04-13T22:37:17.529-07:00 [Info] bucket3/idx1_kzXfpjTh/Mainstore#12218195536005622715:0 Plasma: SMR reclaim pending is higher than expected: pending = 304 KB (expected = 217 KB), wCtxCnt = 9, objCnt 16, changed reclaimList flush threshold from 0 to 1, changed reclaimSize flush threshold from 19 KB to 19 KB.

2021-04-13T22:37:17.550-07:00 [Info] bucket1/idx1_mRAd/Mainstore#15329649308689168382:0 Plasma: SMR reclaim pending is higher than expected: pending = 637 KB (expected = 217 KB), wCtxCnt = 8, objCnt 24, changed reclaimList flush threshold from 2 to 0, changed reclaimSize flush threshold from 21 KB to 21 KB.

2021-04-13T22:37:17.559-07:00 [Info] bucket3/idx1_EyDS/Mainstore#5528891594313300595:0 Plasma: SMR reclaim pending is higher than expected: pending = 318 KB (expected = 217 KB), wCtxCnt = 8, objCnt 21, changed reclaimList flush threshold from 0 to 1, changed reclaimSize flush threshold from 21 KB to 21 KB.

2021-04-13T22:37:17.864-07:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.97.216:58606.  Error = EOF. Kill Pipe.

2021-04-13T22:37:18.118-07:00 [Info] messageListener.start(): message channel closed. Remove peer 1f:f9:ff:a8:d2:2f:2b:87 as follower.

2021-04-13T22:37:18.164-07:00 [Error] panic in Follower.startListener() : runtime error: invalid memory address or nil pointer dereference

2021-04-13T22:37:18.174-07:00 [Error] 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:169 +0x22

github.com/couchbase/gometa/protocol.(*Follower).startListener.func1(0xc091b36230)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/follower.go:133 +0xdb

panic(0x10d18c0, 0x20b7790)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2

github.com/couchbase/indexing/secondary/manager/client.(*watcher).updateIndexStats2NoLock(0xc0142ab3b0, 0xc0c74c9940, 0x20, 0xc0c20c4b68, 0x0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:5435 +0x240

github.com/couchbase/indexing/secondary/manager/client.(*watcher).processChange(0xc0142ab3b0, 0x10000c5b3, 0x7, 0xc0acbf17d0, 0xb, 0xc0f431d600, 0x2bf, 0x2c0, 0xc0c74c9940, 0x20, ...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:6130 +0x904

github.com/couchbase/indexing/secondary/manager/client.(*watcher).Commit(0xc0142ab3b0, 0x10000c5b3, 0x0, 0x0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:5914 +0x21e

github.com/couchbase/gometa/protocol.(*Follower).handleCommit(0xc091b36230, 0x14d3a80, 0xc0d32d9ef0, 0x14d3a80, 0xc0d32d9ef0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/follower.go:255 +0x18c

github.com/couchbase/gometa/protocol.(*Follower).handleMessage(0xc091b36230, 0x14ce420, 0xc0d32d9ef0, 0x14ce420, 0xc0d32d9ef0)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/follower.go:190 +0x253

github.com/couchbase/gometa/protocol.(*Follower).startListener(0xc091b36230)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/follower.go:152 +0x187

created by github.com/couchbase/gometa/protocol.(*Follower).Start

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/follower.go:76 +0x3f

2021-04-13T22:37:18.174-07:00 [Error] WatcherServer.runOnce() : Watcher terminated unexpectedly.

2021-04-13T22:37:18.174-07:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.107.4:9100.  Error = read tcp 172.23.107.2:53942->172.23.107.4:9100: use of closed network connection. Kill Pipe.

2021-04-13T22:37:18.072-07:00 [Info] janitor: running cleanup.

...

...

2021-04-13T22:37:23.583-07:00 [Error] panic in Follower.startListener() : runtime error: invalid memory address or nil pointer dereference

...

...

2021-04-13T22:37:29.682-07:00 [Error] panic in Follower.startListener() : runtime error: invalid memory address or nil pointer dereference