Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.0.3-7031 -> 7.1.0-2373
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Script to Repro
1. Run CC longevity on 7.0.3 for 2-3 days
./sequoia -client 172.23.104.254:2375 -provider file:centos_second_cluster.yml -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml -scale 3 -repeat 0 -log_level 0 -version 7.0.3-7031 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Change the encryption level to strict.
3. Add 2 7.1 kv nodes and 2 7.1 indexing nodes, Remove 2 7.0.3 kv and 2 7.0.3 indexing nodes and do swap rebalance.
Few day after we started the rebalance on the newly added 7.1 kv node I noticed this exit.
172.23.106.138 10:52:04 AM 27 Feb, 2022
Service 'goxdcr' exited with status 2. Restarting. Messages:
|
github.com/couchbase/goxdcr/utils.(*Utilities).QueryRestApiWithAuth(0xc000301a60, 0xc00843c630, 0x13, 0x105571f, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/utils/utils.go:2062 +0x2eb
|
github.com/couchbase/goxdcr/peerToPeer.(*P2pCommAPIimpl).P2PSend(0xc000b15a40, 0x11accc8, 0xc00085a820, 0xc0030dda40, 0x0, 0x0, 0xc000698000)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/peerToPeer/peerToPeerCommAPI.go:133 +0x253
|
github.com/couchbase/goxdcr/peerToPeer.(*P2PManagerImpl).sendToSpecifiedPeersOnce.func1.1(0xc003f99e90, 0xc002bca740, 0xc000a371b8, 0x14, 0xc00843c630, 0x13, 0xc000140280, 0x2, 0xc0030dda40, 0xc006c76c48, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/peerToPeer/peerToPeerManager.go:300 +0x334
|
created by github.com/couchbase/goxdcr/peerToPeer.(*P2PManagerImpl).sendToSpecifiedPeersOnce.func1
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/peerToPeer/peerToPeerManager.go:279 +0x399
|
172.23.106.138
[ns_server:info,2022-02-27T10:52:04.257-08:00,babysitter_of_ns_1@cb.local:goxdcr-goport<0.212.0>:goport:handle_process_exit:566]Port exited with status 2.
|
[error_logger:error,2022-02-27T10:52:04.277-08:00,babysitter_of_ns_1@cb.local:<0.206.0>:ale_error_logger_handler:do_log:101]
|
=========================ERROR REPORT=========================
|
** Generic server <0.206.0> terminating
|
** Last message in was {<0.212.0>,{exit_status,2}}
|
** When Server state == {state,<0.212.0>,114997,
|
{goxdcr,"/opt/couchbase/bin/goxdcr",
|
["-sourceKVAdminPort=8091",
|
"-xdcrRestPort=9998","-isEnterprise=true",
|
"-ipv4=required","-ipv6=off",
|
"-caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem"],
|
[via_goport,exit_status,stderr_to_stdout,
|
{env,
|
[{"GOTRACEBACK","single"},
|
{"CBAUTH_REVRPC_URL",
|
"http://%40:12b38146055ee069005d9c95f948138c@127.0.0.1:8091/goxdcr"}]}]},
|
{ringbuffer,1983,1024,
|
{[],
|
[{<<"panic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x50ffc9]\n\n"...>>,
|
1983}]}},
|
goxdcr,undefined,[],0}
|
** Reason for termination ==
|
** {abnormal,2}
|
cbcollect_info attached.