Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35119

[System Test] XDCR "Execution timed out" errors in replication

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.5.0
    • 6.5.0
    • XDCR
    • 6.5.0-3748

    Description

      Noticed that a replication is in pending state for ~2 hours during which 10 instances of "Execution timed out" are displayed in the UI. Is this expected behavior?

      2019-07-16T11:48:42.531-07:00 INFO GOXDCR.RuntimeCtx: Executing Action timed out
      2019-07-16T11:52:29.324-07:00 INFO GOXDCR.GenericPipeline: Executing Action timed out
      .
      .
      2019-07-16T13:54:42.225-07:00 INFO GOXDCR.GenericPipeline: Executing Action timed out
      2019-07-16T13:56:17.341-07:00 INFO GOXDCR.RuntimeCtx: Executing Action timed out

      2019-07-16T13:15:03.383-07:00 INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: dd5879dc10975cc6fa45589313df1136/default/default InternalId: 7d-WoV_pFvQUoT8ffeDhog== SourceBucketName: default SourceBucketUUID: bfb3389c9da23d7f562b702cb21ace90 TargetClusterUUID: dd5879dc10975cc6fa45589313df1136 TargetBucketName: default TargetBucketUUID: 64c93cfc537d64a4d6c758b1de1ee359 Settings: map[filterExpiration:false failure_restart_interval:10 filter_expression: priority:High backlogThreshold:50 optimistic_replication_threshold:256 worker_batch_size:500 target_nozzle_per_node:2 log_level:Info source_nozzle_per_node:2 filter_expression_version:0 doc_batch_size_kb:2048 stats_interval:1000 active:true bandwidth_limit:0 replication_type:xmem filterBypassExpiry:false checkpoint_interval:600 compression_type:3 filter_exp_del:0 filterDeletion:false filter_skip_restream:false], source bucket uuid=bfb3389c9da23d7f562b702cb21ace90
      		 
      2019-07-16T13:15:03.446-07:00 INFO GOXDCR.PipelineMgr: Replication Status = map[dd5879dc10975cc6fa45589313df1136/default/default:name={dd5879dc10975cc6fa45589313df1136/default/default}, status={Pending}, errors={[]}, oldProgress={Pipeline has been constructed}, progress={Pipeline is constructed}
      		 
      ]
      		 
      2019-07-16T13:15:04.117-07:00 [INFO] Using plain authentication for user <ud>Administrator</ud>
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.RuntimeCtx: Executing Action timed out
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.RuntimeCtx: ****************************
      		 
      2019-07-16T13:15:04.271-07:00 ERRO GOXDCR.RuntimeCtx: dd5879dc10975cc6fa45589313df1136/default/default error starting pipeline context. err=Execution timed out
      		 
      2019-07-16T13:15:04.271-07:00 ERRO GOXDCR.GenericPipeline: dd5879dc10975cc6fa45589313df1136/default/default-242891642 failed to start, err=map[genericPipeline.context.Start:Execution timed out]
      		 
      2019-07-16T13:15:04.271-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the pipeline dd5879dc10975cc6fa45589313df1136/default/default-242891642
      		 
      2019-07-16T13:15:04.271-07:00 ERRO GOXDCR.PipelineMgr: Failed to update pipeline dd5879dc10975cc6fa45589313df1136/default/default, err=genericPipeline.context.Start : Execution timed out
      		 
      2019-07-16T13:15:04.271-07:00 ERRO GOXDCR.PipelineMgr: Update of pipeline dd5879dc10975cc6fa45589313df1136/default/default failed with errors=genericPipeline.context.Start : Execution timed out
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.PipelineMgr: Replication dd5879dc10975cc6fa45589313df1136/default/default update experienced error(s): genericPipeline.context.Start : Execution timed out. Scheduling a redo.
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) genericPipeline.context.Start : Execution timed out, current status=name={dd5879dc10975cc6fa45589313df1136/default/default}, status={Pending}, errors={[{"time":"2019-07-16T13:15:04.271584907-07:00","errMsg":"Execution timed out"}]}, oldProgress={Pipeline is constructed}, progress={Pipeline failed to start, err=map[genericPipeline.context.Start:Execution timed out]}
      		 
       
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.PipelineMgr: Replication dd5879dc10975cc6fa45589313df1136/default/default's status experienced changes or errors (<nil>). However, last update resulted in failure, so will reschedule a future update
      		 
      2019-07-16T13:15:04.271-07:00 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled to update in 10s

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-37819 Remove/Suppress XDCR warnings for non-Data Service nodes

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-35136 TopologyChangeDetector Start() process improvements

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              neil.huang Neil Huang
              pavithra.mahamani Pavithra Mahamani (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty