Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49559

Investigate error observed in longevity with p2p enabled - Pipeline did not start in a timely manner, possibly due to busy source or target. Will try again...

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • Major
    • 7.1.0
    • 7.1.0
    • XDCR
    • 1

    Description

      Following error observed multiple times in our longevity run with p2p enabled:

      2021-11-15 10:44:27 172.23.96.243:Pipeline did not start in a timely manner, possibly due to busy source or target. Will try again...

      From the xdcr logs of the same node:

      021-11-15T10:40:33.577-08:00 WARN GOXDCR.BucketTopologySvc: Goxdcr BucketTopologyWatcher default Error getting high seqno for kv 172.23.96.243:11210. err=Can't find high seqno for vbno=154 in stats map. Source topology may have changed
      		 
      2021-11-15T10:40:33.577-08:00 ERRO GOXDCR.BucketTopologySvc: BucketTopologySvcWatcher for local? true bucket default updating resulted in err Can't find high seqno for vbno=154 in stats map. Source topology may have changed - bypassing notification
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.RuntimeCtx: Executing Action timed out
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.RuntimeCtx: ****************************
      		 
      2021-11-15T10:40:33.803-08:00 ERRO GOXDCR.RuntimeCtx: 644f61320435667363263b3a856ee3e9/bucket8/bucket8 error starting pipeline context. err=Execution timed out
      		 
      2021-11-15T10:40:33.803-08:00 ERRO GOXDCR.GenericPipeline: 644f61320435667363263b3a856ee3e9/bucket8/bucket8-727790848 failed to start, err=map[genericPipeline.context.Start:Execution timed out]
      		 
      2021-11-15T10:40:33.803-08:00 ERRO GOXDCR.PipelineMgr: Failed to start the pipeline 644f61320435667363263b3a856ee3e9/bucket8/bucket8-727790848
      		 
      2021-11-15T10:40:33.803-08:00 ERRO GOXDCR.PipelineMgr: Failed to update pipeline 644f61320435667363263b3a856ee3e9/bucket8/bucket8, err=genericPipeline.context.Start : Execution timed out
      		 
      2021-11-15T10:40:33.803-08:00 ERRO GOXDCR.PipelineMgr: Update of pipeline 644f61320435667363263b3a856ee3e9/bucket8/bucket8 failed with errors=genericPipeline.context.Start : Execution timed out
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.PipelineMgr: Replication 644f61320435667363263b3a856ee3e9/bucket8/bucket8 update experienced error(s): genericPipeline.context.Start : Execution timed out. Scheduling a redo.
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) genericPipeline.context.Start : Execution timed out, current status=name={644f61320435667363263b3a856ee3e9/bucket8/bucket8}, status={Pending}, errors={[{"time":"2021-11-15T10:40:33.803555619-08:00","errMsg":"genericPipeline.context.Start:Execution timed out"},{"time":"2021-11-15T10:37:52.898155196-08:00","errMsg":"RuntimeCtx:Execution timed out"},{"time":"2021-11-15T10:37:32.670742244-08:00","errMsg":"genericPipeline.context.Start:Execution timed out"},{"time":"2021-11-15T10:34:56.228952093-08:00","errMsg":"TopoChangeDet : Restarting pipeline due to target topology change..."}]}, oldProgress={Done PeerToPeer communication and metadata merging}, progress={Pipeline failed to start, err=map[genericPipeline.context.Start:Execution timed out]}
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.PipelineMgr: Replication 644f61320435667363263b3a856ee3e9/bucket8/bucket8's status experienced changes or errors (<nil>). However, last update resulted in failure, so will reschedule a future update
      		 
      2021-11-15T10:40:33.803-08:00 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled to update in 10s

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.106.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collectinfo-2021-11-15T210655-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1637010414/collect

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-58539 XDCR - ckptMgr stop can hang if start was not called

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-58545 [BP 7.2.1] - XDCR - ckptMgr stop can hang if start was not called

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-58546 [7.1.5-MP] - XDCR - ckptMgr stop can hang if start was not called

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MB-49552 XDCR - P2P - perform ckptPull only on non-existent or expired vb checkpoints

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-49377 XDCR - P2P Re-enablement plan

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty