Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[Magma] - Cleaning up of the cluster fails with "Rebalance exited with reason {buckets_shutdown_wait_failed"

Description

Script to Repro

172.23.120.206 10:05:03 PM 11 Nov, 2021 ( 2021-11-11T22:05:03.228-08:00 )

Even retried rebalance failed.
172.23.120.206 10:06:28 PM 11 Nov, 2021

Based on the failures it does look like the previously dropped bucket too longer than expected to get deleted.

172.23.104.186 10:03:32 PM 11 Nov, 2021

Maybe we need to figure out a way to disable the rebalance button until the bucket is fully deleted.

cbcollect_info attached.

Components

Affects versions

Fix versions

Labels

Environment

Enterprise Edition 7.1.0 build 1694 ‧

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Attachments

7

Activity

Show:

CB robot March 21, 2022 at 1:25 PM

Build couchbase-server-7.2.0-1021 contains kv_engine commit 0517833 with commit message:
: Make Taskable::isShutdown() const

CB robot March 21, 2022 at 1:25 PM

Build couchbase-server-7.2.0-1021 contains kv_engine commit f74b76b with commit message:
: Reset task ptr on scheduler thread during taskable shutdown

Pavithra Mahamani March 16, 2022 at 6:00 PM

Closing, will file a new ticket based on 's comment that the root cause is not the same for the latest occurrence.

Ben Huddleston March 15, 2022 at 8:00 AM

Looks to be a different root cause for this issue .

The cleanup in kv_engine looks to be prompt, but the shutdown of the magma shards is rather slow taking multiple minutes. , probably worth somebody on the magma team taking a look at the latest set of logs.

Pavithra Mahamani March 15, 2022 at 3:01 AM
Edited

I am still seeing this issue in 7.1.0-2475 in the XDCR functional tests.

Logs:

,

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

No

Triage

Triaged

Operating System

Centos 64-bit

Due date

Story Points

Parent

Sprint

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created November 12, 2021 at 6:18 AM
Updated March 21, 2022 at 1:25 PM
Resolved March 16, 2022 at 6:00 PM
Instabug