Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48677

[System Test][XDCR] RuntimeCtx : Execution timed out - observed in longevity during topology changes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.1.0
    • 7.1.0
    • XDCR
    • Untriaged
    • 1
    • Unknown

    Description

      7.1.0-1377

      Test:
      -test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
      Scale 3
      Iteration 1

      .97.121:

      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.XDCRRouter: CollectionsRouter Router_dcp_8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4_172.23.97.121:11210_1 is now stopping
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.XDCRRouter: CollectionsRouter Router_dcp_8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4_172.23.97.121:11210_1 is now stopping
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.XDCRRouter: Router Router_dcp_8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4_172.23.97.121:11210_1 stopped
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.GenericPipeline: 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4-29121742 pipeline connectors have stopped successfully
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.GenericPipeline: MainPipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4-29121742 has been stopped
      		 
       errMap=map[RuntimeCtx:Execution timed out]
      		 
      *2021-09-28T23:35:36.212-07:00 ERRO GOXDCR.PipelineMgr: Received error(s) when stopping pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4 - RuntimeCtx : Execution timed out*
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.PipelineMgr: Replication Status=name={8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4}, status={Pending}, errors={[{"time":"2021-09-28T23:35:31.009864849-07:00","errMsg":"TopoChangeDet : Restarting pipeline due to source topology change..."}]}, oldProgress={Source nozzles have been closed}, progress={Pipeline has been stopped}, oldBackfillProgress={Source nozzles have been closed}, backfillProgress={Pipeline has been stopped}
      		 
      2021-09-28T23:35:36.212-07:00 ERRO GOXDCR.PipelineMgr: Failed to update pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4, err=RuntimeCtx : Execution timed out
      		 
      2021-09-28T23:35:36.212-07:00 ERRO GOXDCR.PipelineMgr: Update of pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4 failed with errors=RuntimeCtx : Execution timed out
      		 
      2021-09-28T23:35:36.212-07:00 INFO GOXDCR.PipelineMgr: Pipeline updater scheduled to update in 10s
      		 
      2021-09-28T23:35:36.352-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket9/bucket9 total_docs=5859, docs_processed=5166, changes_left=693
      		 
      2021-09-28T23:35:36.809-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/default/remote total_docs=290394, docs_processed=254552, changes_left=35842
      		 
      2021-09-28T23:35:37.166-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 total_docs=3906, docs_processed=3444, changes_left=462
      		 
      2021-09-28T23:35:37.192-07:00 INFO GOXDCR.TopoChangeDet: TopologyChangeDetectorSvc for pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 handleTargetTopologyChange completed
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: Source topology has changed for pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: Pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 - vblist_supposed=[113 114 115 116 117 118 119 120 121 429 430 431 432 433 434 435 436 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 522 523], vblist_now=[429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513]
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: Source topology changed for pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8: vblist_removed=[437 438 439], vblist_new=[113 114 115 116 117 118 119 120 121 522 523]
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: Number of source topology changes seen by pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 is 23
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: Number of consecutive stable source topology seen by pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 is 22
      		 
      2021-09-28T23:35:37.202-07:00 INFO GOXDCR.TopoChangeDet: TopologyChangeDetectorSvc for pipeline 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 handleSourceTopologyChange completed
      		 
      2021-09-28T23:35:37.362-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket9/bucket9 total_docs=5859, docs_processed=5166, changes_left=693
      		 
      2021-09-28T23:35:37.445-07:00 INFO GOXDCR.CheckpointMgr: Aborting checkpointing routine for MainPipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4 with vb list [429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513] since received finish signal. index=66
      		 
      2021-09-28T23:35:37.445-07:00 INFO GOXDCR.CheckpointMgr: MainPipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4 Exits checkpointing routine.
      		 
      2021-09-28T23:35:37.445-07:00 INFO GOXDCR.RuntimeCtx: 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4 successfully stopped service CheckpointManager.
      		 
      2021-09-28T23:35:37.471-07:00 INFO GOXDCR.CheckpointMgr: Done checkpointing for replication MainPipeline 8e4dc755a84c1b076a3bc60e00223311/bucket4/bucket4
      		 
      2021-09-28T23:35:37.845-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/default/remote total_docs=290747, docs_processed=254932, changes_left=35815
      		 
      2021-09-28T23:35:38.165-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 total_docs=3906, docs_processed=3444, changes_left=462
      		 
      2021-09-28T23:35:38.351-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket9/bucket9 total_docs=5859, docs_processed=5166, changes_left=693
      		 
      2021-09-28T23:35:38.405-07:00 WARN GOXDCR.CheckpointMgr: 8e4dc755a84c1b076a3bc60e00223311/default/remote Failed to find snapshot for vb=496, seqno=1394
      		 
       
      		 
      2021-09-28T23:35:38.809-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/default/remote total_docs=291156, docs_processed=255258, changes_left=35898
      		 
      2021-09-28T23:35:39.163-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 total_docs=3906, docs_processed=3444, changes_left=462
      		 
      2021-09-28T23:35:39.351-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket9/bucket9 total_docs=5859, docs_processed=5166, changes_left=693
      		 
      2021-09-28T23:35:39.811-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/default/remote total_docs=291512, docs_processed=255608, changes_left=35904
      		 
      2021-09-28T23:35:40.163-07:00 INFO GOXDCR.StatsMgr: 8e4dc755a84c1b076a3bc60e00223311/bucket8/bucket8 total_docs=3906, docs_processed=3444, changes_left=462
      		 
      2021-09-28T23:35:40.294-07:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 174 highTP: 174 highExist: true lowExist: false backlogExist: true maxTP: 174 highTPNeeded: 1140940 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 20 idleCpu: 77
      		 
      2021-09-28T23:35:40.294-07:00 INFO GOXDCR.ResourceMgr: backlogCount=636, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632953128/collectinfo-2021-09-29T220531-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-48787 XDCR - backfill pipeline not stopping all async listeners

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48722 XDCR - async listener may not stop properly

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty