Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48860

[System Test] Service 'goxdcr' exited with status 137. Execution timed out.

    XMLWordPrintable

Details

    Description

      Test -

      -test tests/integration/neo/test_neo_magma_wo_gsi_n1ql.yml -scope tests/integration/neo/scope_neo_magma_wo_gsi_n1ql.yml

      Day - 1
      Scale - 3
      Cycle - 4

      172.23.120.77
      diag.log

      2021-10-11T19:00:04.681-07:00, ns_log:0:info:message(ns_1@172.23.123.24) - Service 'goxdcr' exited with status 137. Restarting. Messages:
      		 
      2021-10-11T18:59:58.451-07:00 WARN GOXDCR.Utils: GetBucketInfo(172.23.123.33:8091, bucket8) took 1m2.05527319s
      		 
      2021-10-11T18:59:55.513-07:00 WARN GOXDCR.CheckpointMgr: Unable to find Target manifest ID for vb 401
      		 
      2021-10-11T18:59:58.474-07:00 WARN GOXDCR.Utils: Get(ckpt/4c1b34dca75c61d454790b89688f66b8/default/remote/266) took 7.962599543s
      		 
      2021-10-11T18:59:58.499-07:00 WARN GOXDCR.Utils: Set(ckpt/4c1b34dca75c61d454790b89688f66b8/default/remote/807) took 5.551806815s
      		 
      2021-10-11T18:59:58.098-07:00 INFO GOXDCR.HttpServer: Executing Action timed out
      		 
      2021-10-11T18:59:58.612-07:00 INFO GOXDCR.HttpServer: ****************************
      		 
      2021-10-11T18:59:58.612-07:00 ERRO GOXDCR.HttpServer: Execution timed out
      		 
      2021-10-11T18:59:58.612-07:00 ERRO GOXDCR.HttpServer: Execution timed out
      		 
      2021-10-11T18:59:58.059-07:00 ERRO GOXDCR.HttpServer: Execution timed out
      		 
      2021-10-11T18:59:58.730-07:00 ERRO GOXDCR.HttpServer: Execution timed out
      		 
      2021-10-11T18:59:59.004-07:00 WARN GOXDCR.Utils: Get(ckpt/4c1b34dca75c61d454790b89688f66b8/bucket8/bucket8/259) took 3.286723597s

      ns_server.goxdcr.log
      Following error messages observed around the time of failure.

      2021-10-11T18:55:10.062-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the backfill pipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-654087903
      		 
      2021-10-11T18:55:11.568-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:55:17.179-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:55:17.179-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:55:20.121-07:00 ERRO GOXDCR.CheckpointMgr: BackfillPipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4 has not been started - checkpointing is skipped
      		 
      2021-10-11T18:55:27.187-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:55:27.187-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:55:28.176-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:55:33.402-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 250347520 timed out
      		 
      2021-10-11T18:55:36.264-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc065bc9f20 0xc11d81da00} timed out
      		 
      2021-10-11T18:55:36.501-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc04a74cad0 0xc0d6586b60} timed out
      		 
      2021-10-11T18:55:38.583-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc04d2666a0 0xc0550a7e10} timed out
      		 
      2021-10-11T18:55:38.663-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc086ebfc50 0xc0dfd5ad00} timed out
      		 
      2021-10-11T18:55:39.465-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc086ebfb50 0xc0763b2270} timed out
      		 
      2021-10-11T18:55:40.151-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:55:40.321-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc05dfe2bb0 0xc04c9795f0} timed out
      		 
      2021-10-11T18:55:43.234-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc04d266690 0xc0071be410} timed out
      		 
      2021-10-11T18:55:46.377-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:55:48.603-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3834118144 timed out
      		 
      2021-10-11T18:55:50.832-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc04d266590 0xc0cfe72f70} timed out
      		 
      2021-10-11T18:55:55.352-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:04.145-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1792278528 timed out
      		 
      2021-10-11T18:56:07.778-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:10.752-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:19.953-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1156513792 timed out
      		 
      2021-10-11T18:56:25.154-07:00 ERRO GOXDCR.GenericPipeline: backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-761628085 failed to start, err=map[genericPipeline.RunP2PProtocol:Execution timed out]
      		 
      2021-10-11T18:56:25.154-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the backfill pipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-761628085
      		 
      2021-10-11T18:56:26.553-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:35.193-07:00 ERRO GOXDCR.CheckpointMgr: BackfillPipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4 has not been started - checkpointing is skipped
      		 
      2021-10-11T18:56:35.587-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:36.568-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3762028544 timed out
      		 
      2021-10-11T18:56:39.567-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3a3500950 0xc198827d40} timed out
      		 
      2021-10-11T18:56:43.158-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:48.277-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2083979264 timed out
      		 
      2021-10-11T18:56:52.823-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc2df7bad70 0xc08feaab60} timed out
      		 
      2021-10-11T18:56:54.754-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 356384768 timed out
      		 
      2021-10-11T18:56:55.748-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3339ae6e0 0xc14fa3d5f0} timed out
      		 
      2021-10-11T18:56:55.748-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc34707d3e0 0xc0341784e0} timed out
      		 
      2021-10-11T18:56:55.748-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc34707d4e0 0xc09758f520} timed out
      		 
      2021-10-11T18:56:56.139-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3339ae5e0 0xc2ccf25520} timed out
      		 
      2021-10-11T18:56:56.606-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:56:56.868-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc2eadf4da0 0xc0844881a0} timed out
      		 
      2021-10-11T18:56:58.283-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc2df7bab70 0xc093ed7380} timed out
      		 
      2021-10-11T18:57:01.359-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:03.477-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 771162112 timed out
      		 
      2021-10-11T18:57:04.914-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc34707d2e0 0xc0d6586340} timed out
      		 
      2021-10-11T18:57:11.806-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:16.155-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2410151936 timed out
      		 
      2021-10-11T18:57:16.188-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:16.189-07:00 ERRO GOXDCR.P2PManager: Unable to send VBMasterCheck to some or all the nodes... sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:16.189-07:00 ERRO GOXDCR.P2PManager: sendToEachPeerOnce err sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:17.171-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:57:17.171-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:57:18.878-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1268711424 timed out
      		 
      2021-10-11T18:57:22.760-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:27.210-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:57:27.210-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:57:27.211-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:34.678-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2341011456 timed out
      		 
      2021-10-11T18:57:41.606-07:00 ERRO GOXDCR.GenericPipeline: backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-278350018 failed to start, err=map[genericPipeline.RunP2PProtocol:Execution timed out]
      		 
      2021-10-11T18:57:41.606-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the backfill pipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-278350018
      		 
      2021-10-11T18:57:43.111-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:43.957-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3722706944 timed out
      		 
      2021-10-11T18:57:50.561-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:57:51.280-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2093416448 timed out
      		 
      2021-10-11T18:57:51.607-07:00 ERRO GOXDCR.CheckpointMgr: BackfillPipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4 has not been started - checkpointing is skipped
      		 
      2021-10-11T18:57:59.714-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:01.685-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1407713280 timed out
      		 
      2021-10-11T18:58:03.638-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc1bbafc720 0xc0489f81a0} timed out
      		 
      2021-10-11T18:58:09.481-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3451977728 timed out
      		 
      2021-10-11T18:58:11.253-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc234d10550 0xc071ed88f0} timed out
      		 
      2021-10-11T18:58:11.549-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc234d10450 0xc04ad05c70} timed out
      		 
      2021-10-11T18:58:13.506-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc234d10050 0xc1b7aa48f0} timed out
      		 
      2021-10-11T18:58:13.719-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc292a133e0 0xc1426e80d0} timed out
      		 
      2021-10-11T18:58:14.152-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc234d10350 0xc20b1ef930} timed out
      		 
      2021-10-11T18:58:14.758-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3553d3ce0 0xc08e19e5b0} timed out
      		 
      2021-10-11T18:58:15.147-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:16.885-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1094057984 timed out
      		 
      2021-10-11T18:58:17.915-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:20.491-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3553d3ec0 0xc1d27b8000} timed out
      		 
      2021-10-11T18:58:24.556-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1089011712 timed out
      		 
      2021-10-11T18:58:27.536-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3553d3ae0 0xc2731ca4e0} timed out
      		 
      2021-10-11T18:58:30.347-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:30.881-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 4146397184 timed out
      		 
      2021-10-11T18:58:31.161-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:31.162-07:00 ERRO GOXDCR.P2PManager: Unable to send VBMasterCheck to some or all the nodes... sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:31.162-07:00 ERRO GOXDCR.P2PManager: sendToEachPeerOnce err sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:32.359-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1810432000 timed out
      		 
      2021-10-11T18:58:39.316-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:45.748-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:58:48.161-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2528641024 timed out
      		 
      2021-10-11T18:58:58.683-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3973447680 timed out
      		 
      2021-10-11T18:59:00.134-07:00 ERRO GOXDCR.GenericPipeline: backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-580254023 failed to start, err=map[genericPipeline.RunP2PProtocol:Execution timed out]
      		 
      2021-10-11T18:59:00.134-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the backfill pipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-580254023
      		 
      2021-10-11T18:59:01.549-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:04.785-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 4008902656 timed out
      		 
      2021-10-11T18:59:07.141-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:10.143-07:00 ERRO GOXDCR.CheckpointMgr: BackfillPipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4 has not been started - checkpointing is skipped
      		 
      2021-10-11T18:59:17.186-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:59:17.186-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:59:17.657-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3090612224 timed out
      		 
      2021-10-11T18:59:18.152-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:27.194-07:00 ERRO GOXDCR.CollectionsManifestSvc: Specified entity is not found
      		 
      2021-10-11T18:59:27.194-07:00 ERRO GOXDCR.CollectionsManifestSvc: Error with callback: Specified entity is not found
      		 
      2021-10-11T18:59:27.314-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc05dfe2cb0 0xc0fafe3450} timed out
      		 
      2021-10-11T18:59:29.424-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2007367680 timed out
      		 
      2021-10-11T18:59:30.146-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3e0e36300 0xc00659b110} timed out
      		 
      2021-10-11T18:59:30.248-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3e0e36500 0xc017456d00} timed out
      		 
      2021-10-11T18:59:32.665-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc03afa6ac0 0xc05367a9c0} timed out
      		 
      2021-10-11T18:59:32.858-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2758213632 timed out
      		 
      2021-10-11T18:59:33.454-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3bb74faf0 0xc130d37790} timed out
      		 
      2021-10-11T18:59:33.895-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:35.507-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc03afa6c90 0xc02d9d0b60} timed out
      		 
      2021-10-11T18:59:35.585-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3e18fcdd0 0xc1174cd6c0} timed out
      		 
      2021-10-11T18:59:36.353-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:39.284-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 386269184 timed out
      		 
      2021-10-11T18:59:42.174-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc03afa6e90 0xc0dd812b60} timed out
      		 
      2021-10-11T18:59:47.741-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:47.741-07:00 ERRO GOXDCR.P2PManager: Unable to send VBMasterCheck to some or all the nodes... sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:47.741-07:00 ERRO GOXDCR.P2PManager: sendToEachPeerOnce err sendPeerToPeerReq(VBMasterCheck) Operation failed after max retries.  Last error: 172.23.123.24:8091 : Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:48.260-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 2134048768 timed out
      		 
      2021-10-11T18:59:49.136-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T18:59:50.826-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1691025408 timed out
      		 
      2021-10-11T18:59:52.343-07:00 ERRO GOXDCR.P2PManager: Unable to respond to caller given {0xc3bb74f8f0 0xc038d08270} timed out
      		 
      2021-10-11T18:59:57.768-07:00 ERRO GOXDCR.P2PManager: P2PSend resulted in Post "http://172.23.123.24:8091/xdcr/p2pCommunications": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      		 
      2021-10-11T19:00:04.176-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 4101963776 timed out
      		 
      2021-10-11T19:00:18.633-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 1930821632 timed out
      		 
      2021-10-11T19:00:18.895-07:00 ERRO GOXDCR.GenericPipeline: backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-782465220 failed to start, err=map[genericPipeline.RunP2PProtocol:Execution timed out]
      		 
      2021-10-11T19:00:18.895-07:00 ERRO GOXDCR.PipelineMgr: Failed to start the backfill pipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4-782465220
      		 
      2021-10-11T19:00:20.811-07:00 ERRO GOXDCR.P2PManager: Request type 0 to 172.23.123.24:8091 with opaque 3868852224 timed out
      		 
      2021-10-11T19:00:22.579-07:00 ERRO GOXDCR.P2PManager: Unable to send PeriodicPush to some or all the nodes... sendPeerToPeerReq(PeriodicPush) Operation failed after max retries.  Last error: 172.23.123.25:8091 : <nil>-500
      		 
      2021-10-11T19:00:28.896-07:00 ERRO GOXDCR.CheckpointMgr: BackfillPipeline backfill_4c1b34dca75c61d454790b89688f66b8/bucket4/bucket4 has not been started - checkpointing is skipped

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-48850 [System Test] XDCR OOM killed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-48884 [System Test] Few cluster operations failed due to error - Internal server error

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-49085 XDCR - statsMgr for backfill pipeline may accumulate notifications

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-48888 XDCR - upgrade golang to 1.16 or later to address large heapIdle issues

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-48889 XDCR - utilize sync.Pool for bucketTopologySvc notifications

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty