Prevent import cfg startup races

Description

The import DCP feed (run by cbgt) manages its cfg with a CfgSg instance, which stores the cfg files as documents in the database's backing bucket.  CfgSg uses the caching DCP feed to subscribe to changes to those files.

Starting the import DCP feed before the caching DCP feed can result in missed notifications about cfg changes, as the caching DCP feed starts from the high vbseq values. 

In addition, the heartbeater may triggers changes to the import's cfg files when checking for stale heartbeats (which by default occurs for the first time 2s after the heartbeater is initialized).  If the cfg change triggered by stale heartbeat detection occurs before the caching DCP feed is started, cbgt won't be notified of the change to remove the stale node.

Two changes should be made during databaseContext creation:

  • start the caching DCP feed prior to the import DCP feed

  • move heartbeater.StartCheckingHeartbeats to the end of dbContext initialization, to ensure all other components are initialized before doing any stale node cleanup.  Note that StartSendingHeartbeats still needs to be called prior to cfg initialization, at the start of NewDatabaseContext, so this will require splitting heartbeater.Start into two separate functions

Activity

Show:

CB robot January 7, 2021 at 12:11 PM

Build sync_gateway-3.0.0-97 contains sync_gateway commit a9e22a1 with commit message:
Avoid DCP and cfg race on database start (#4903)

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Components

Sprint

Fix versions

Affects versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created December 16, 2020 at 1:36 AM
Updated August 31, 2024 at 11:00 AM
Resolved January 7, 2021 at 12:44 PM
Instabug