[MB-41255] Value eviction bucket - Replication of deletes with XATTRS will fail if return EWOULD_BLOCK - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 6.5.0, 6.0.3, 6.0.4, 6.5.1, 6.6.0
Fix Version/s: 6.0.5, 6.6.1, 6.5.2
Component/s: couchbase-bucket
Labels:

Triage:
Triaged
Story Points:
1
Is this a Regression?:
Yes

Description

Problem

As described in CBSE-8828, for a replica DCP stream. If a previous item that uses XATTRs is to be deleted, however it is not resident - then we must perform a bgfetch - see ~~MB-36087~~. When the bgfetch is complete we replay PassiveStream::messageReceived. However this time it fails because the last seqno now equals the sequence number of the deletion and so it is discarded.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

image001.png
90 kB
04/Sep/20 8:48 AM
image001.png
90 kB
04/Sep/20 8:34 AM
image001.png
90 kB
04/Sep/20 4:22 AM

Issue Links

backports to

MB-41387 [BP] MB-41255 to 6.0.4 - MP2

Resolved

blocks

MB-41310 6.0.4-MP2

Resolved

is caused by

MB-36087 DeleteWithMeta against a value evicted xattr document crashes memcached

Closed

relates to

MB-41471 Upgrading to MB-41255

Resolved

mentioned in: Page Loading...

Sub-Tasks

[BP] MB-41255 to 6.0.4 - MP2

Resolved

Daniel Owen

Gerrit Reviews

- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Daniel Owen created issue - 01/Sep/20 3:58 AM

Daniel Owen made changes - 01/Sep/20 3:59 AM

Field	Original Value	New Value
Link		This issue relates to CBSE-8828 [ CBSE-8828 ]

Daniel Owen made changes - 01/Sep/20 4:00 AM

Link

This issue relates to ~~MB-36087~~ [ ~~MB-36087~~ ]

Daniel Owen made changes - 01/Sep/20 4:00 AM

Fix Version/s

6.6.1 [ 17002 ]

Daniel Owen made changes - 01/Sep/20 4:00 AM

Is this a Regression?

Unknown [ 10452 ]

Yes [ 10450 ]

Dave Rigby added a comment - 01/Sep/20 4:02 AM

only set the last seqno to the seqno of the message just received, one the message has been completed or

If an EWOULD_BLOCK occurs reserve the setting of the last seqno so the message can be received again.

The former makes most sense to me - note that EWOULDBLOCK can in theory happen mutliple times (say we are unlucky and it gets evicted before the request is scheduled again.

Dave Rigby added a comment - 01/Sep/20 4:02 AM only set the last seqno to the seqno of the message just received, one the message has been completed or If an EWOULD_BLOCK occurs reserve the setting of the last seqno so the message can be received again. The former makes most sense to me - note that EWOULDBLOCK can in theory happen mutliple times (say we are unlucky and it gets evicted before the request is scheduled again.

Daniel Owen made changes - 01/Sep/20 4:02 AM

Triage

Untriaged [ 10351 ]

Triaged [ 10350 ]

Daniel Owen made changes - 01/Sep/20 4:03 AM

Summary

Replication of deletes with XATTRS can fail due to EWOULD_BLOCK

Replication of deletes with XATTRS will fail if return EWOULD_BLOCK

Daniel Owen made changes - 01/Sep/20 4:04 AM

Description

+Problem+

As described in CBSE-8828, for a replica DCP stream. If a previous item that uses XATTRs is to be deleted, however it is not resident - then we must perform a bgfetch - see ~~MB-36087~~. When the bgfetch is complete we replay PassiveStream::messageReceived. However this time it fails because the last seqno now equals the sequence number of the deletion and so it is discarded.

+Suggested Possible Solution+

- only set the last seqno to the seqno of the message just received, one the message has been completed or
- If an EWOULD_BLOCK occurs reserve the setting of the last seqno so the message can be received again.

+Problem+

As described in CBSE-8828, for a replica DCP stream. If a previous item that uses XATTRs is to be deleted, however it is not resident - then we must perform a bgfetch - see ~~MB-36087~~. When the bgfetch is complete we replay PassiveStream::messageReceived. However this time it fails because the last seqno now equals the sequence number of the deletion and so it is discarded.

+Suggested Possible Solution+

- only set the last seqno to the seqno of the message just received, once the message has been completed or
- If an EWOULD_BLOCK occurs reserve the setting of the last seqno so the message can be received again.

Daniel Owen made changes - 01/Sep/20 8:58 AM

Priority

Major [ 3 ]

Critical [ 2 ]

Dave Rigby made changes - 02/Sep/20 3:20 AM

Rank

Ranked higher

Dave Rigby made changes - 02/Sep/20 3:24 AM

Link

This issue is caused by ~~MB-36087~~ [ ~~MB-36087~~ ]

Dave Rigby made changes - 02/Sep/20 3:24 AM

Link

This issue relates to ~~MB-36087~~ [ ~~MB-36087~~ ]

Daniel Owen made changes - 03/Sep/20 3:49 AM

Assignee

Daniel Owen [ owend ]

Jim Walker [ jwalker ]

Jim Walker added a comment - 03/Sep/20 5:11 AM - edited

The fix is actually inside deleteWithMeta - when that function is invoked for a replica delete, it shouldn't be trying to 'prune' xattrs (i.e. need to bg-fetch existing data). As per comments already in that function, when invoked for a replica, just take what you're given and store it.

The issue here is that a clause earlier in the function triggers the bg-fetch for either replica or not

Jim Walker added a comment - 03/Sep/20 5:11 AM - edited The fix is actually inside deleteWithMeta - when that function is invoked for a replica delete, it shouldn't be trying to 'prune' xattrs (i.e. need to bg-fetch existing data). As per comments already in that function, when invoked for a replica, just take what you're given and store it. The issue here is that a clause earlier in the function triggers the bg-fetch for either replica or not

Jim Walker made changes - 03/Sep/20 6:17 AM

Status

Open [ 1 ]

In Progress [ 3 ]

Perry Krug made changes - 04/Sep/20 4:22 AM

Attachment

image001.png [ 106493 ]

Perry Krug made changes - 04/Sep/20 8:34 AM

Attachment

image001.png [ 106524 ]

Perry Krug made changes - 04/Sep/20 8:48 AM

Attachment

image001.png [ 106531 ]

James Flather made changes - 04/Sep/20 8:57 AM

Comment

[ A comment with security level 'Membase Inc' was removed. ]

James Flather made changes - 04/Sep/20 8:57 AM

Comment

[ A comment with security level 'Membase Inc' was removed. ]

James Flather made changes - 04/Sep/20 8:57 AM

Comment

[ A comment with security level 'Membase Inc' was removed. ]

James Flather made changes - 04/Sep/20 8:57 AM

Security

Private [ 10010 ]

Wayne Siu made changes - 04/Sep/20 11:01 AM

Link

This issue blocks ~~MB-41310~~ [ ~~MB-41310~~ ]

Wayne Siu made changes - 04/Sep/20 11:11 AM

Link

This issue blocks MB-40528 [ MB-40528 ]

Wayne Siu made changes - 04/Sep/20 11:11 AM

Labels

approved-for-6.0.5 approved-for-6.6.1

Wayne Siu made changes - 04/Sep/20 11:12 AM

Link

This issue blocks MB-40511 [ MB-40511 ]

James Flather made changes - 07/Sep/20 4:08 AM

Link

This issue relates to CBSP-3173 [ CBSP-3173 ]

Couchbase Build Team added a comment - 07/Sep/20 7:48 AM

Build couchbase-server-6.6.1-9038 contains kv_engine commit 8c73162 with commit message:
~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 07/Sep/20 7:48 AM Build couchbase-server-6.6.1-9038 contains kv_engine commit 8c73162 with commit message: MB-41255 : Don't bgfetch for a replica delete of an xattr

Ian McCloy made changes - 07/Sep/20 8:09 AM

Security

Private [ 10010 ]

Couchbase Build Team added a comment - 09/Sep/20 1:09 AM

Build couchbase-server-7.0.0-3058 contains kv_engine commit 8c73162 with commit message:
~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 09/Sep/20 1:09 AM Build couchbase-server-7.0.0-3058 contains kv_engine commit 8c73162 with commit message: MB-41255 : Don't bgfetch for a replica delete of an xattr

Jim Walker added a comment - 09/Sep/20 4:53 AM

Merged to mad-hatter (for 6.6.1)
Forward merged to master

Jim Walker added a comment - 09/Sep/20 4:53 AM Merged to mad-hatter (for 6.6.1) Forward merged to master

Jim Walker made changes - 09/Sep/20 4:53 AM

Assignee	Jim Walker [ jwalker ]	Daniel Owen [ owend ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

Daniel Owen made changes - 09/Sep/20 5:17 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

Daniel Owen made changes - 10/Sep/20 7:34 AM

Resolution	Fixed [ 1 ]
Status	Closed [ 6 ]	Reopened [ 4 ]

Couchbase Build Team added a comment - 11/Sep/20 6:35 AM

Build couchbase-server-6.0.4-3098 contains kv_engine commit 829f660 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 11/Sep/20 6:35 AM Build couchbase-server-6.0.4-3098 contains kv_engine commit 829f660 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 11/Sep/20 9:23 AM

Build couchbase-server-6.0.5-3302 contains kv_engine commit 829f660 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 11/Sep/20 9:23 AM Build couchbase-server-6.0.5-3302 contains kv_engine commit 829f660 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Wayne Siu made changes - 11/Sep/20 5:35 PM

Fix Version/s

6.0.5 [ 16729 ]

Wayne Siu made changes - 15/Sep/20 3:37 PM

Remote Link

This issue links to "Page (Couchbase, Inc. Wiki)" [ 20577 ]

Wayne Siu made changes - 15/Sep/20 4:25 PM

Link

This issue blocks MB-41489 [ MB-41489 ]

Daniel Owen added a comment - 16/Sep/20 3:54 AM

Hey Jim Walker Could you merge the patch into the 6.5.1 branch that Chris Hillery created? (see MB-41489) thanks

Daniel Owen added a comment - 16/Sep/20 3:54 AM Hey Jim Walker Could you merge the patch into the 6.5.1 branch that Chris Hillery created? (see MB-41489) thanks

Daniel Owen made changes - 16/Sep/20 3:54 AM

Assignee

Daniel Owen [ owend ]

Jim Walker [ jwalker ]

Daniel Owen made changes - 16/Sep/20 5:00 AM

Link

This issue relates to ~~MB-41471~~ [ ~~MB-41471~~ ]

Jim Walker made changes - 18/Sep/20 4:20 AM

Summary

Replication of deletes with XATTRS will fail if return EWOULD_BLOCK

Value eviction bucket - Replication of deletes with XATTRS will fail if return EWOULD_BLOCK

Couchbase Build Team added a comment - 18/Sep/20 8:10 AM

Build couchbase-server-6.5.1-6303 contains kv_engine commit 5958386 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 18/Sep/20 8:10 AM Build couchbase-server-6.5.1-6303 contains kv_engine commit 5958386 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 21/Sep/20 2:31 AM

Build couchbase-server-7.0.0-3163 contains kv_engine commit a242708 with commit message:
~~MB-41255~~: Create an improve unit test that has more coverage

Couchbase Build Team added a comment - 21/Sep/20 2:31 AM Build couchbase-server-7.0.0-3163 contains kv_engine commit a242708 with commit message: MB-41255 : Create an improve unit test that has more coverage

Jim Walker made changes - 28/Sep/20 5:35 AM

Assignee	Jim Walker [ jwalker ]	Daniel Owen [ owend ]
Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Daniel Owen made changes - 01/Oct/20 2:07 AM

Link

This issue backports to ~~MB-41387~~ [ ~~MB-41387~~ ]

Couchbase Build Team added a comment - 01/Oct/20 4:22 AM

Build couchbase-server-6.6.1-9110 contains kv_engine commit 829f660 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 01/Oct/20 4:22 AM Build couchbase-server-6.6.1-9110 contains kv_engine commit 829f660 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 02/Oct/20 6:56 PM

Build couchbase-server-6.5.1-6307 contains kv_engine commit 5958386 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 02/Oct/20 6:56 PM Build couchbase-server-6.5.1-6307 contains kv_engine commit 5958386 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Daniel Owen made changes - 26/Oct/20 6:34 AM

Link

This issue relates to CBSE-9100 [ CBSE-9100 ]

Couchbase Build Team added a comment - 04/Nov/20 8:00 AM

Build couchbase-server-7.0.0-3630 contains kv_engine commit 829f660 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 04/Nov/20 8:00 AM Build couchbase-server-7.0.0-3630 contains kv_engine commit 829f660 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Wayne Siu made changes - 06/Nov/20 6:53 PM

Link

This issue blocks MB-42583 [ MB-42583 ]

Jim Walker added a comment - 11/Nov/20 1:55 AM

To reproduce the orginal issue something like the following is required.

At least 2 nodes (1 value eviction bucket) are needed as we need replication.
Bucket should have a very low quota to make this quicket/easier to trigger.
Fill the bucket with items that have xattrs (this bit requires a client that can create items with xattrs).
- The bucket become less than 100% resident
- The items that we write in this stage will later need to be deleted, so it maybe convenient to set an expiry value, but if you record the keys that are written it is more reliably to explicitly delete them later.
Once the bucket has a good number of non-resident items start deleting the items (or wait for the expiry interval and force compaction)
The issue is reproduced once one of the items is deleted and replicated, provided that on the replica the item is not resident the bug occurs.

Jim Walker added a comment - 11/Nov/20 1:55 AM To reproduce the orginal issue something like the following is required. At least 2 nodes (1 value eviction bucket) are needed as we need replication. Bucket should have a very low quota to make this quicket/easier to trigger. Fill the bucket with items that have xattrs (this bit requires a client that can create items with xattrs). The bucket become less than 100% resident The items that we write in this stage will later need to be deleted, so it maybe convenient to set an expiry value, but if you record the keys that are written it is more reliably to explicitly delete them later. Once the bucket has a good number of non-resident items start deleting the items (or wait for the expiry interval and force compaction) The issue is reproduced once one of the items is deleted and replicated, provided that on the replica the item is not resident the bug occurs.

Ashwin Govindarajulu added a comment - 16/Nov/20 10:56 PM

Thanks for providing the validation steps Jim Walker

Validated the fix using 6.6.1-9182 build.

Closing this ticket

Ashwin Govindarajulu added a comment - 16/Nov/20 10:56 PM Thanks for providing the validation steps Jim Walker Validated the fix using 6.6.1-9182 build. Closing this ticket

Ashwin Govindarajulu made changes - 16/Nov/20 10:56 PM

Assignee	Daniel Owen [ owend ]	Ashwin Govindarajulu [ ashwin.govindarajulu ]
Status	Resolved [ 5 ]	Closed [ 6 ]

Wayne Siu made changes - 01/Dec/20 7:28 PM

Fix Version/s

6.5.2 [ 17223 ]

Abhishek Jindal made changes - 15/Dec/20 9:57 PM

Link

This issue relates to CBSE-9361 [ CBSE-9361 ]

Ashwin Govindarajulu added a comment - 18/Dec/20 3:07 AM

Validated the fix against "6.0.5-3333-enterprise"

http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1480/console

Ashwin Govindarajulu added a comment - 18/Dec/20 3:07 AM Validated the fix against "6.0.5-3333-enterprise" http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1480/console

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM

Build couchbase-server-6.6.2-9411 contains kv_engine commit c690fba with commit message:
Merge "~~MB-41255~~: Merge branch '6.5.2' into mad-hatter" into mad-hatter

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM Build couchbase-server-6.6.2-9411 contains kv_engine commit c690fba with commit message: Merge " MB-41255 : Merge branch '6.5.2' into mad-hatter" into mad-hatter

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM

Build couchbase-server-6.6.2-9411 contains kv_engine commit d88694a with commit message:
~~MB-41255~~: Merge branch '6.5.2' into mad-hatter

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM Build couchbase-server-6.6.2-9411 contains kv_engine commit d88694a with commit message: MB-41255 : Merge branch '6.5.2' into mad-hatter

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM

Build couchbase-server-6.6.2-9411 contains kv_engine commit 5958386 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 04/Jan/21 6:42 AM Build couchbase-server-6.6.2-9411 contains kv_engine commit 5958386 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Arunkumar Senthilnathan (Inactive) made changes - 12/Jan/21 1:39 PM

Labels

approved-for-6.0.5 approved-for-6.6.1

approved-for-6.0.5 approved-for-6.6.1 releasenote

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM

Build couchbase-server-7.0.0-4255 contains kv_engine commit c690fba with commit message:
Merge "~~MB-41255~~: Merge branch '6.5.2' into mad-hatter" into mad-hatter

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM Build couchbase-server-7.0.0-4255 contains kv_engine commit c690fba with commit message: Merge " MB-41255 : Merge branch '6.5.2' into mad-hatter" into mad-hatter

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM

Build couchbase-server-7.0.0-4255 contains kv_engine commit d88694a with commit message:
~~MB-41255~~: Merge branch '6.5.2' into mad-hatter

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM Build couchbase-server-7.0.0-4255 contains kv_engine commit d88694a with commit message: MB-41255 : Merge branch '6.5.2' into mad-hatter

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM

Build couchbase-server-7.0.0-4255 contains kv_engine commit 5958386 with commit message:
[BP] ~~MB-41255~~: Don't bgfetch for a replica delete of an xattr

Couchbase Build Team added a comment - 19/Jan/21 6:59 AM Build couchbase-server-7.0.0-4255 contains kv_engine commit 5958386 with commit message: [BP] MB-41255 : Don't bgfetch for a replica delete of an xattr

Shivani Gupta added a comment - 20/Oct/21 9:21 AM

Thanks Dave Rigby.

Depends on how you define "regression" . I believe the issue was introduced with XATTR support; when replicating deletes to items which previously had XATTRs in them. To hit this issue you need to be using XATTRs, which were new in 5.0.0 - so while it's true we didn't encounter this issue prior to v5, you also need to be using the new v5 feature to hit it.

Customer was upgrading from 6.0 to 6.6. Given that the fix was backported to 6.0.5 also, looks like the issue existed in 6.0.x as well. Likely that the customer was not using XATTRs and their previous upgrades (to 6.0.x) were hence not impacted. So yes, probably not a regression.

Shivani Gupta added a comment - 20/Oct/21 9:21 AM Thanks Dave Rigby . Depends on how you define "regression" . I believe the issue was introduced with XATTR support; when replicating deletes to items which previously had XATTRs in them. To hit this issue you need to be using XATTRs, which were new in 5.0.0 - so while it's true we didn't encounter this issue prior to v5, you also need to be using the new v5 feature to hit it. Customer was upgrading from 6.0 to 6.6. Given that the fix was backported to 6.0.5 also, looks like the issue existed in 6.0.x as well. Likely that the customer was not using XATTRs and their previous upgrades (to 6.0.x) were hence not impacted. So yes, probably not a regression.