Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60021

[XDCR][Windows] : Service 'goxdcr' exited with status 3221226505

    XMLWordPrintable

Details

    Description

      1. Created a 6 node cluster with index, kv, n1ql service nodes
      2. Created 2 couchbase buckets named "default" and "standard_bucket"
      3. Started a workload using cbworkloadgen 

      Observing goxdcr crash and restarts recurringly throughout the run

      2023-12-03T02:09:50.368-08:00, ns_log:0:info:message(ns_1@172.23.137.55) - Service 'goxdcr' exited with status 3221226505. Restarting. Messages:2023-12-03T02:09:32.622-08:00 INFO GOXDCR.P2PManager: Successfully sent request type Discovery to nodes with opaque: map[172.23.137.56:8091:861143040 172.23.137.57:8091:3503292416]2023-12-03T02:09:32.623-08:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.137.57:80912023-12-03T02:09:35.368-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]2023-12-03T02:09:36.811-08:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.137.57:80912023-12-03T02:09:36.823-08:00 INFO GOXDCR.P2PManager: Discovered peers: map[172.23.137.56:8091:EmsUaKLhTv-dkgzchBKzKY4hprn-D8Sul0gmObHF3e4= 172.23.137.57:8091:UHcvN9CE1GNMh9Gjwi3GXU9rC78xQO8257-OQuGJBfE=]2023-12-03T02:09:40.364-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = <nil>2023-12-03T02:09:40.364-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=02023-12-03T02:09:40.364-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]ongoingReplMap=map[]

      Observing this multiple times in ns_server.goxdcr.log

      2023-12-03T02:03:20.264-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.59:8091 : key being added already exists172.23.137.55:8091 : key being added already exists172.23.137.58:8091 : key being added already exists). Sleeping 200ms2023-12-03T02:03:20.265-08:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.137.56:80912023-12-03T02:03:20.470-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.55:8091 : key being added already exists172.23.137.59:8091 : key being added already exists). Sleeping 400ms2023-12-03T02:03:20.471-08:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.137.58:80912023-12-03T02:03:20.875-08:00 ERRO GOXDCR.P2PManager: Failed to set authentication to request. err=Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:20.875-08:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:20.875-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.55:8091 : key being added already exists172.23.137.59:8091 : Unable to find given hostport in cbauth database: `172.23.137.59:8091'). Sleeping 800ms2023-12-03T02:03:21.689-08:00 ERRO GOXDCR.P2PManager: Failed to set authentication to request. err=Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:21.689-08:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:21.689-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.55:8091 : key being added already exists172.23.137.59:8091 : Unable to find given hostport in cbauth database: `172.23.137.59:8091'). Sleeping 1.6s2023-12-03T02:03:23.222-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]2023-12-03T02:03:23.296-08:00 ERRO GOXDCR.P2PManager: Failed to set authentication to request. err=Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:23.296-08:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:23.296-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.55:8091 : key being added already exists172.23.137.59:8091 : Unable to find given hostport in cbauth database: `172.23.137.59:8091'). Sleeping 3.2s2023-12-03T02:03:24.334-08:00 INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type Discovery from 172.23.137.56:80912023-12-03T02:03:24.342-08:00 INFO GOXDCR.P2PManager: Discovered peers: map[172.23.137.55:8091:q3Zc21p0oOW52bgxm9M8oo1nqVRSDxwFgHgM2UZv014= 172.23.137.56:8091:Nv2uo7M-BMmRLXhisJiAlV2Ao_TknKCCnf67WX9kuwQ= 172.23.137.58:8091:LKS3ivXq6jVljTtdnzUPiAzgN7RV2MdhszQEK6JZ5ZE=]2023-12-03T02:03:26.509-08:00 ERRO GOXDCR.P2PManager: Failed to set authentication to request. err=Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:26.509-08:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Unable to find given hostport in cbauth database: `172.23.137.59:8091'2023-12-03T02:03:26.509-08:00 WARN GOXDCR.Utils: ExponentialBackoffExecutor for sendPeerToPeerReq(Discovery) encountered error (172.23.137.55:8091 : key being added already exists172.23.137.59:8091 : Unable to find given hostport in cbauth database: `172.23.137.59:8091'). Sleeping 6.4s

       

      The n1ql panics and crashes are tracked under MB-60020

      Not sure if they are related

       

       

       

      Testrunner script to repro

       

      ./testrunner -i /tmp/testexec.21806.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False,get-cbcollect-info=True,get-cbcollect-info=True -t clitest.workloadgentest.workloadgenTests.workloadgen_test,replica=0,standard_buckets=1,itmes=1000000,command_options=-j -t 2,GROUP=P0

       

      Job name : windows19-os_certify-tools_1

      Job ref : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-1878/jenkins_logs/test_suite_executor-dynvm/18439/

       

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-59031 xdcr_changes_left_total in Windows XDCR tests can't reach 0 for hours

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-59928 goxdcr crashes repeatedly due to js-evaluator

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty