[System Test][XDCR] ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: xx actual count xx - observed in longevity

2021-10-05T17:35:18.338-07:00 ERRO GOXDCR.Adminport: Unable to generate req or resp Invalid input given

2021-10-05T17:35:18.425-07:00 ERRO GOXDCR.ThrSeqTrackSvc: ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]

2021-10-05T17:35:18.425-07:00 ERRO GOXDCR.GenericSupervisor: PipelineSupervisor_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 Received error report : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]

. errors_seen=map[backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker:ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]]

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.ReplMgr: Supervisor PipelineSupervisor_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 of type *supervisor.GenericSupervisor reported errors map[backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker:ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]]

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: PipelineOpSerializer 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 handling job: Job for 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 Type: PipelineUpdate 

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) r.update_err_ch : backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254], current status=name={5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4}, status={Replicating}, errors={[{"time":"2021-10-05T17:35:18.425742054-07:00","errMsg":"backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]"}]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}, oldBackfillProgress={Pipeline is running}, backfillProgress={Received error report : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]}

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: PipelineOpSerializer 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 done handling job: Job for 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4 Type: PipelineUpdate 

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: Replication 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4's status experienced changes or errors (r.update_err_ch : backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254]), updating now

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4. Current error(s)=r.update_err_ch : backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker : ThroughSeqno backfill_5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4_ThroughSeqnoTracker background waiting backfill streams to all be replicated timed out. Expected count: 85 actual count 63. VBs waiting on: [243 168 165 207 161 242 304 768 169 803 767 167 650 166 240 244 483 491 649 112 176 254] 

2021-10-05T17:35:18.425-07:00 INFO GOXDCR.PipelineMgr: Try to stop pipeline 5f4b32de7a13b85221a2eb842ed656ea/bucket4/bucket4