hit panic: runtime error: invalid memory address or nil pointer dereference on build 7.1.6-3955

Test: Avg. initial XDCR rate (items/sec), 1 -> 2 changes to 2 -> 2 (2 source nozzles, 4 target nozzles), 1 bucket x 200M x 1KB

Build: 7.1.6-3955

Job: https://perf.jenkins.couchbase.com/job/titan_5node2/2628/

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-2628/172.23.96.105.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-2628/172.23.96.106.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-2628/172.23.96.107.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-2628/172.23.96.108.zip

2023-11-06T06:44:22 [ERROR] Following failures found: { "panics":

{ "./172.23.96.106.zip": [ "cbcollect_info_ns_1@172.23.96.106_20231106-144024/ns_server.goxdcr.log" ] }

}

 

2023-11-06T06:39:36.722-08:00 INFO GOXDCR.BackfillMgr: Starting backfill request handler for spec e441452851ffb68ec3f63834d32bfaf7/bucket-1/bucket-1 internalId dRPI7Se6kLucLTPv2RsbrA==
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8f055a]
 
goroutine 168 [running]:
github.com/couchbase/goxdcr/peerToPeer.(*P2PManagerImpl).ReplicationSpecChangeCallback(0xc0001b5a40, {0xc00004c7e1?, 0x32?}, {0xf77ae0?, 0x0?}, {0xf77ae0?, 0xc00071ebd0?}, 0xc000741301?)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/peerToPeer/peerToPeerManager.go:584 +0x29a
github.com/couchbase/goxdcr/metadata_svc.(*ReplicationSpecService).executeCallbackWithPriority(0xc0006460f0, {0xc00004c7e1, 0x32}, 0xc0006386c0?, 0x1194bc0?, 0x0, 0x1)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/replication_spec_service.go:1337 +0x163
github.com/couchbase/goxdcr/metadata_svc.(*ReplicationSpecService).callMetadataChangeCb(0xc0006460f0, {0xc00004c7e1, 0x32}, 0xc00071ebd0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/replication_spec_service.go:1300 +0x1d6
github.com/couchbase/goxdcr/metadata_svc.(*ReplicationSpecService).updateCacheInternal(0xc0006460f0, {0xc00004c7e1, 0x32}, 0xc00071ebd0, 0x0?)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/replication_spec_service.go:1205 +0x3bb
github.com/couchbase/goxdcr/metadata_svc.(*ReplicationSpecService).updateCache(...)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/replication_spec_service.go:1169
github.com/couchbase/goxdcr/metadata_svc.(*ReplicationSpecService).ReplicationSpecServiceCallback(0xc0006460f0, {0xc00004c7d0, 0x43}, {0xc0006da800, 0x7c6, 0x7c8}, {0xe76600, 0xc00042c7f8})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/metadata_svc/replication_spec_service.go:1132 +0x215
github.com/couchbase/goxdcr/replication_manager.(*MetakvChangeListener).metakvCallback_async(0xc000494900, {0xc00004c7d0?, 0xe861a0?}, {0xc0006da800?, 0x40cbe5?, 0xc0006b69c0?}, {0xe76600?, 0xc00042c7f8?})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:188 +0x5f
github.com/couchbase/goxdcr/replication_manager.(*callbackSerializer).handleJobs(0xc00011ba70, {0xc00004c7d0, 0x43})
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:110 +0x1ed
created by github.com/couchbase/goxdcr/replication_manager.(*callbackSerializer).distributeJob
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/replication_manager/metakv_change_listener.go:86 +0x2d5