[MB-35889] Replication can get stuck when checkpoint memory overhead is very high - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 6.5.0, 6.5.1
Fix Version/s: 6.6.0
Component/s: couchbase-bucket
Labels:

Triage:
Untriaged
Epic Link:
KV: Magma
Is this a Regression?:
No
Sprint:
KV-Engine MH 2nd Beta, KV Sprint 2020-April

Description

Build 6.5.0-4218

Observed that replication stuck when data service goes into low resident ratio.
While running some HiDD tests on couchbase bucket we came across this issue.
In this test we have 2 data nodes, load 250M docs and RR goes to 0.43%. After load phase we wait for "ep_dcp_replica_items_remaining" to go to zero. "ep_dcp_replica_items_remaining" stays ~19K and never become zero.

Job- http://perf.jenkins.couchbase.com/job/magma-hidd/441
Logs-
https://cb-jira.s3.us-east-2.amazonaws.com/logs/replica_issue_couchbase/collectinfo-2019-09-10T055022-ns_1%40172.23.97.38.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/replica_issue_couchbase/collectinfo-2019-09-10T055022-ns_1%40172.23.97.39.zip

Attachments

Issue Links

causes

MB-39435 [Jepsen] Multiple crashes during Checkpoint::expelItems() due to Expects() throwing

Closed

is duplicated by

MB-39440 Magma cbc-pillowfight tests stuck during the load phase

Closed

relates to

MB-41283 Crash in checkpoint code due to keyIndexes pointing to freed queued_items

Closed

MB-38012 1% DGM Test: Write ops/s dropped from 230k/s to 400/s due to swapping

Closed

MB-35970 Investigate freeing keyIndex when closing checkpoint

Resolved

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Mahesh Mandhare (Inactive) created issue - 10/Sep/19 2:37 AM

Raju Suravarjjala made changes - 10/Sep/19 10:19 AM

Field	Original Value	New Value
Fix Version/s		Mad-Hatter [ 15037 ]

Sarath Lakshman made changes - 10/Sep/19 6:03 PM

Labels

Performance

Performance hidd

Daniel Owen made changes - 11/Sep/19 12:52 AM

Assignee

Daniel Owen [ owend ]

Jim Walker [ jwalker ]

Jim Walker made changes - 17/Sep/19 1:28 AM

Assignee	Jim Walker [ jwalker ]	Mahesh Mandhare [ mahesh.mandhare ]
Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Jim Walker made changes - 17/Sep/19 1:54 AM

Assignee	Mahesh Mandhare [ mahesh.mandhare ]	Jim Walker [ jwalker ]
Resolution	Fixed [ 1 ]
Status	Resolved [ 5 ]	Reopened [ 4 ]

Jim Walker made changes - 17/Sep/19 8:18 AM

Status

Reopened [ 4 ]

In Progress [ 3 ]

Dave Rigby made changes - 18/Sep/19 12:29 AM

Sprint

KV-Engine MH 2nd Beta [ 872 ]

Dave Rigby made changes - 18/Sep/19 12:29 AM

Rank

Ranked higher

James Harrison made changes - 19/Sep/19 8:45 AM

Link

This issue is duplicated by ~~MB-35970~~ [ ~~MB-35970~~ ]

Jim Walker made changes - 25/Sep/19 2:10 AM

Assignee	Jim Walker [ jwalker ]	Mahesh Mandhare [ mahesh.mandhare ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

Mahesh Mandhare (Inactive) made changes - 01/Oct/19 3:38 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

Mahesh Mandhare (Inactive) made changes - 04/Oct/19 11:07 PM

Resolution	Fixed [ 1 ]
Status	Closed [ 6 ]	Reopened [ 4 ]

Mahesh Mandhare (Inactive) made changes - 05/Oct/19 4:29 AM

Assignee

Mahesh Mandhare [ mahesh.mandhare ]

Jim Walker [ jwalker ]

Jim Walker made changes - 07/Oct/19 2:13 AM

Fix Version/s		Cheshire-Cat [ 15915 ]
Fix Version/s	Mad-Hatter [ 15037 ]

Dave Rigby made changes - 09/Oct/19 1:55 AM

Sprint

KV-Engine MH 2nd Beta [ 872 ]

KV-Engine MH 2nd Beta, KV-Engine MH 2nd Beta 2 [ 872, 910 ]

Dave Rigby made changes - 25/Oct/19 1:30 AM

Sprint

KV-Engine MH 2nd Beta, KV-Engine Mad-Hatter GA [ 872, 910 ]

KV-Engine MH 2nd Beta [ 872 ]

Dave Rigby made changes - 25/Oct/19 1:30 AM

Rank

Ranked higher

Daniel Owen made changes - 02/Jan/20 5:44 AM

Epic Link

~~MB-30659~~ [ 88207 ]

Ben Huddleston made changes - 04/Mar/20 7:30 AM

Assignee

Jim Walker [ jwalker ]

Ben Huddleston [ ben.huddleston ]

Ben Huddleston made changes - 04/Mar/20 7:40 AM

Link

This issue relates to ~~MB-38012~~ [ ~~MB-38012~~ ]

Ben Huddleston made changes - 06/Mar/20 6:51 AM

Summary

Replication stuck if data service in low resident ratio

Replication can get stuck when checkpoint memory overhead is very high

Ben Huddleston made changes - 10/Mar/20 4:44 AM

Sprint

KV-Engine MH 2nd Beta [ 872 ]

KV-Engine MH 2nd Beta, KV Spint 2020-March [ 872, 1002 ]

Ben Huddleston made changes - 10/Mar/20 4:44 AM

Rank

Ranked lower

Daniel Owen made changes - 17/Apr/20 6:19 AM

Sprint

KV-Engine MH 2nd Beta, KV Spint 2020-March [ 872, 1002 ]

KV-Engine MH 2nd Beta, KV Sprint 2020-April [ 872, 1044 ]

Daniel Owen made changes - 17/Apr/20 6:19 AM

Rank

Ranked lower

Daniel Owen made changes - 30/Apr/20 7:14 AM

Link

This issue relates to CBSE-8284 [ CBSE-8284 ]

Dave Rigby made changes - 07/May/20 1:31 PM

Labels

Performance hidd

Performance candidate-for-6.6 hidd

Dave Rigby made changes - 07/May/20 1:32 PM

Fix Version/s

6.6.0 [ 16787 ]

Ben Huddleston made changes - 13/May/20 1:49 AM

Link

This issue is duplicated by ~~MB-35970~~ [ ~~MB-35970~~ ]

Ben Huddleston made changes - 13/May/20 1:49 AM

Link

This issue relates to ~~MB-35970~~ [ ~~MB-35970~~ ]

Daniel Owen made changes - 13/May/20 3:18 AM

Affects Version/s

6.5.1 [ 16622 ]

Dave Rigby made changes - 14/May/20 6:40 AM

Priority

Major [ 3 ]

Critical [ 2 ]

Till Westmann made changes - 17/May/20 11:15 PM

Link

This issue blocks MB-38724 [ MB-38724 ]

Till Westmann made changes - 17/May/20 11:15 PM

Labels

Performance candidate-for-6.6 hidd

Performance approved-for-6.6.0 candidate-for-6.6 hidd

Dave Rigby made changes - 18/May/20 1:43 AM

Fix Version/s	Cheshire-Cat [ 15915 ]
Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Ben Huddleston made changes - 18/May/20 2:55 AM

Resolution	Fixed [ 1 ]
Status	Resolved [ 5 ]	Reopened [ 4 ]

Ben Huddleston made changes - 18/May/20 11:24 PM

Link

This issue causes ~~MB-39435~~ [ ~~MB-39435~~ ]

Ben Huddleston made changes - 19/May/20 3:01 AM

Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Wayne Siu made changes - 19/May/20 7:01 PM

Assignee

Ben Huddleston [ ben.huddleston ]

Bo-Chun Wang [ bo-chun.wang ]

Ben Huddleston made changes - 20/May/20 1:33 AM

Link

This issue is duplicated by ~~MB-39440~~ [ ~~MB-39440~~ ]

Wayne Siu made changes - 20/May/20 5:56 PM

Labels

Performance approved-for-6.6.0 candidate-for-6.6 hidd

Performance affects-cc-testing approved-for-6.6.0 candidate-for-6.6 hidd

Bo-Chun Wang made changes - 21/Jul/20 2:03 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

Richard deMellow made changes - 16/Sep/20 12:47 AM

Link

This issue relates to ~~MB-41283~~ [ ~~MB-41283~~ ]

People

Assignee:: Bo-Chun Wang

Reporter:: Mahesh Mandhare (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: 10/Sep/19 2:37 AM

Updated:: 16/Sep/20 12:47 AM

Resolved:: 19/May/20 3:01 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 13 closed Gerrit changes

Hide There are 13 closed Gerrit changes

MB-35889: Purge Checkpoint key indexes during expel and state change: Gerrit Review:

MB-35889: Use tracking allocator for Checkpoint memOverhead tracking: Gerrit Review:

MB-35889: Tune the dcp cursor dropping tests: Gerrit Review:

MB-36301: Revert "MB-35889: Tune the dcp cursor dropping tests": Gerrit Review:

MB-36301: Partial Revert "MB-35889: Purge Checkpoint key indexes during expel and state change": Gerrit Review:

MB-36301: Revert "MB-35889: Use tracking allocator for Checkpoint memOverhead tracking": Gerrit Review:

Revert "MB-36301: Partial Revert "MB-35889: Purge Checkpoint key indexes during expel and state change"": Gerrit Review:

MB-35889: Expel committed items from disk checkpoints: Gerrit Review:

MB-35889: Drop checkpoint key indexes on checkpoint close: Gerrit Review:

MB-35889: Drop checkpoint key indexes on checkpoint close: Gerrit Review:

MB-35889: Don't add keys to Checkpoint indexes for Disk checkpoints: Gerrit Review:

MB-35889: Don't invalidate index entry for Disk checkpoint for expel: Gerrit Review:

Merge branch 'mad-hatter': Gerrit Review:

Couchbase Server

Replication can get stuck when checkpoint memory overhead is very high

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty