[XDCR] P2PSend resulted in Unable to find given hostport in cbauth database

[2022-09-06 16:02:44,883] - [rest_client:1470] INFO - adding remote cluster hostname:172.23.107.241:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.107.243:8091

[2022-09-06 16:02:44,884] - [rest_client:1454] INFO - with settings {'hostname': '172.23.107.241:8091', 'username': 'Administrator', 'password': 'password', 'name': 'remote_cluster_C1-C2'}

[2022-09-06 16:02:44,890] - [rest_client:1072] ERROR - POST http://172.23.107.243:8091/pools/default/remoteClusters body: hostname=172.23.107.241%3A8091&username=Administrator&password=password&name=remote_cluster_C1-C2 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password

[error_logger:error,2022-09-06T16:02:44.068-07:00,babysitter_of_ns_1@cb.local:<0.20129.0>:ale_error_logger_handler:do_log:101]

=========================ERROR REPORT=========================

** Generic server <0.20129.0> terminating

** Last message in was {<0.20135.0>,{exit_status,139}}

** When Server state == {state,<0.20135.0>,14443,

                            {goxdcr,"/opt/couchbase/bin/goxdcr",

                                ["-sourceKVAdminPort=8091",

                                 "-xdcrRestPort=9998","-isEnterprise=true",

                                 "-ipv4=required","-ipv6=optional",

                                 "-caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem"],

                                [via_goport,exit_status,stderr_to_stdout,

                                 {env,

                                     [{"GOTRACEBACK","single"},

                                      {"GOMAXPROCS","4"},

                                      {"CBAUTH_REVRPC_URL",

                                       "http://%40:346c4a3cf620d10f3d0c94187b8ee553@127.0.0.1:8091/goxdcr"}]}]},

                            {ringbuffer,1063,1024,

                                {[{<<"2022-09-06T16:02:37.845-07:00 INFO GOXDCR.P2PManager: Discovered peers: map[172.23.107.242:8091:lEFCGdWMAy5g87GNDldnVIBknbZuwLFerkGBlGuRwxA=]\n">>,

                                   142},

                                  {<<"2022-09-06T16:02:37.845-07:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.107.242:8091\n">>,

                                   120},

                                  {<<"2022-09-06T16:02:34.086-07:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false back"...>>,

                                   529}],

                                 [{<<"2022-09-06T16:02:29.126-07:00 INFO GOXDCR.P2PManager: Successfully sent request type Discovery to nodes with opaque: map[172.23.107.242:8091:623"...>>,

                                   152},

                                  {<<"2022-09-06T16:02:29.130-07:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.107.242:8091\n">>,

                                   120}]}},

                            goxdcr,undefined,[],0}

** Reason for termination ==

** {abnormal,139}

[error_logger:error,2022-09-06T16:02:44.069-07:00,babysitter_of_ns_1@cb.local:<0.20129.0>:ale_error_logger_handler:do_log:101]

=========================CRASH REPORT=========================

  crasher:

    initial call: ns_port_server:init/1

    pid: <0.20129.0>

    registered_name: []

    exception exit: {abnormal,139}

      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 811)

    ancestors: [<0.20128.0>,<0.20127.0>,ns_child_ports_sup,

                  ns_babysitter_sup,<0.111.0>]

    message_queue_len: 1

    messages: [{'EXIT',<0.20135.0>,normal}]

    links: [<0.20128.0>]

    dictionary: []

    trap_exit: true

    status: running

    heap_size: 6772

    stack_size: 29

    reductions: 39190

  neighbours:

[ns_server:info,2022-09-06T16:02:44.069-07:00,babysitter_of_ns_1@cb.local:<0.20128.0>:supervisor_cushion:handle_info:54]Cushion managed supervisor for goxdcr failed:  {abnormal,139}

[error_logger:error,2022-09-06T16:02:44.069-07:00,babysitter_of_ns_1@cb.local:<0.20128.0>:ale_error_logger_handler:do_log:101]

2022-09-06T16:02:56.387-07:00 ERRO GOXDCR.P2PManager: Failed to set authentication to request. err=Unable to find given hostport in cbauth database: `172.23.107.242:8091'

2022-09-06T16:02:56.387-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Unable to find given hostport in cbauth database: `172.23.107.242:8091'

2022-09-06T16:02:56.387-07:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.107.242:8091 : Unable to find given hostport in cbauth database: `172.23.107.242:8091'). Sleeping 200ms