[BP 7.1.1] - indexer blocked during storage warmup on MOI storage, causing rebalance failure.

Description

 when indexer is in warmup phase on MOI till the storage warmup is complete,  indexer is blocked from processing other messages, this causes rebalance request to fail. 

 

Components

Affects versions

Fix versions

Labels

Environment

None

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Activity

Show:

CB robot October 7, 2022 at 8:54 AM

Build couchbase-server-7.2.0-5000 contains indexing commit 6f1729a with commit message:
: indexer blocked during storage warmup on MOI storage, causing rebalance failure.

Varun Velamuri July 4, 2022 at 8:59 AM

Closing this based on dev-verification. Steps to repro are mentioned in the above comment

CB robot June 22, 2022 at 11:07 AM

Build couchbase-server-7.1.2-3321 contains indexing commit 6f1729a with commit message:
: indexer blocked during storage warmup on MOI storage, causing rebalance failure.

Varun Velamuri June 22, 2022 at 7:50 AM

Validated the fix using the following steps:

a. Add a sleep of 100 seconds in MemDBSlice::OpenSnapshot method

b. Populate 10k documents in the bucket default

c. create index idx_1 on default(name)

d. Failover indexer node after index build is finished

e. Add back the indexer node and initiate rebalance

 

Before the fix: Rebalance failed with timeout on "PrepareTopologyChange" message

After the fix: Prepare phase goes through and ns_server sends a StartTopologyChange. Rebalance master waits for 10 minutes for indexer to warmup i.e. load the snapshot

 

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

No

Triage

Untriaged

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created April 27, 2022 at 6:21 AM
Updated October 7, 2022 at 8:54 AM
Resolved June 22, 2022 at 7:26 AM
Instabug