[MB-42555] gomemcached does not mark as fatal OS and timeout errors - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 4.0.0, 4.1.2, 4.5.1, 4.6.4, 5.0.1, Cheshire-Cat, 6.0.2, 5.5.5, 6.6.1
Fix Version/s: 6.6.1, 6.5.2, 7.0.0
Component/s: query
Labels:
- approved-for-6.6.1
- releasenote

Triage:
Untriaged
Story Points:
1
Is this a Regression?:
Unknown

Description

Marking it as query, but it affects any component that uses go-couchbase or gomemcached.
When a gomemcached Get() ends prematurely (the most likely cause being a golang timeout),
client/transport.go:getResponse() leaves the response structure uninitialized, which means that it has a 0 status.
This is not picked as fatal, and the connection gets eventually reused, reading incorrectly, data left behind on the wire.

Attachments

Options
- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

Gerrit Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Marco Greco created issue - 06/Nov/20 2:39 AM

Marco Greco made changes - 06/Nov/20 2:41 AM

Field	Original Value	New Value
Business Impact/value	P3 - Nice to have [ 10553 ]
Component/s	query [ 11829 ]
Component/s		query [ 10718 ]
Is this a Regression?		Unknown [ 10452 ]
Key	~~CBSE-9179~~	~~MB-42555~~
Triage		Untriaged [ 10351 ]
Affects Version/s	6.6.1 [ 17140 ]
Affects Version/s	Cheshire-Cat [ 16249 ]
Affects Version/s	6.0.2 [ 16246 ]
Affects Version/s	5.5.5 [ 16245 ]
Affects Version/s	4.6.4 [ 15100 ]
Affects Version/s	5.0.1 [ 15071 ]
Affects Version/s	4.5.1 [ 13613 ]
Affects Version/s	4.1.2 [ 13611 ]
Affects Version/s	4.0.0 [ 12709 ]
Affects Version/s		6.6.1 [ 17002 ]
Affects Version/s		Cheshire-Cat [ 15915 ]
Affects Version/s		4.0.0 [ 12148 ]
Affects Version/s		4.1.2 [ 13312 ]
Affects Version/s		4.5.1 [ 13411 ]
Affects Version/s		4.6.4 [ 14607 ]
Affects Version/s		5.0.1 [ 15052 ]
Affects Version/s		6.0.2 [ 15919 ]
Affects Version/s		5.5.5 [ 16166 ]
Workflow	CBSE workflow [ 184345 ]	classic default workflow [ 184346 ]
Project	Couchbase Support/Engineering Tasks [ 10061 ]	Couchbase Server [ 10010 ]

Marco Greco made changes - 06/Nov/20 2:41 AM

Link

This issue blocks CBSE-9129 [ CBSE-9129 ]

Marco Greco made changes - 06/Nov/20 5:18 AM

Fix Version/s		6.6.1 [ 17002 ]
Fix Version/s		Cheshire-Cat [ 15915 ]

Hide

Permalink

Couchbase Build Team added a comment - 06/Nov/20 5:27 AM

Build couchbase-server-7.0.0-3652 contains gomemcached commit 8e4fcec with commit message:
~~MB-42555~~ mark connections as unusable on golang / os socket errors

Show

Couchbase Build Team added a comment - 06/Nov/20 5:27 AM Build couchbase-server-7.0.0-3652 contains gomemcached commit 8e4fcec with commit message: MB-42555 mark connections as unusable on golang / os socket errors

Raju Suravarjjala made changes - 06/Nov/20 11:35 AM

Link

This issue blocks MB-40528 [ MB-40528 ]

Marco Greco made changes - 06/Nov/20 11:51 AM

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Marco Greco made changes - 06/Nov/20 11:52 AM

Assignee

Marco Greco [ marco.greco ]

Mihir Kamdar [ mihir.kamdar ]

Wayne Siu made changes - 06/Nov/20 11:53 AM

Labels

approved-for-6.6.1

Wayne Siu made changes - 06/Nov/20 11:58 AM

Link

This issue blocks MB-42576 [ MB-42576 ]

Hide

Permalink

Couchbase Build Team added a comment - 06/Nov/20 12:39 PM

Build couchbase-server-6.6.1-9177 contains gomemcached commit 7b3f110 with commit message:
~~MB-42555~~ mark connections as unusable on golang / os socket errors

Show

Couchbase Build Team added a comment - 06/Nov/20 12:39 PM Build couchbase-server-6.6.1-9177 contains gomemcached commit 7b3f110 with commit message: MB-42555 mark connections as unusable on golang / os socket errors

Hide

Permalink

Ajay Bhullar added a comment - 06/Nov/20 12:44 PM

verified in 7.0.0-3652 using sitarams script in the CBSE got a successful repro in 6.5.1 and in 7.0.0-3652 we dont see the bulk get error after running the script 10-15 times, running it twice triggered the issue in 6.5.1

Show

Ajay Bhullar added a comment - 06/Nov/20 12:44 PM verified in 7.0.0-3652 using sitarams script in the CBSE got a successful repro in 6.5.1 and in 7.0.0-3652 we dont see the bulk get error after running the script 10-15 times, running it twice triggered the issue in 6.5.1

Hide

Permalink

Couchbase Build Team added a comment - 06/Nov/20 1:34 PM

Build couchbase-server-6.5.1-6312 contains gomemcached commit 7b3f110 with commit message:
~~MB-42555~~ mark connections as unusable on golang / os socket errors

Show

Couchbase Build Team added a comment - 06/Nov/20 1:34 PM Build couchbase-server-6.5.1-6312 contains gomemcached commit 7b3f110 with commit message: MB-42555 mark connections as unusable on golang / os socket errors

Hide

Permalink

Ajay Bhullar added a comment - 06/Nov/20 2:03 PM

we still see the error in 6.6.1-9177

tid: 0 loop: 22

{"elapsedTime": "1.489946ms", "executionTime": "1.300617ms", "resultSize": 0, "resultCount": 0, "errorCount": 2}

[

{"msg": "Error performing bulk get operation - cause: read tcp 127.0.0.1:47996->127.0.0.1:11210: i/o timeout", "code": 12008, "retry": true}

{"msg": "Timeout 1ms exceeded", "code": 1080, "retry": true}

]

Using sitarams script in the cbse and his repro steps:

VM 8 cpu, 8gb machine

single node cluster 2GB data node
Bucket onebigjson 2GB
Create primary index on onebigjson;
SELECT META(d).id, d.* FROM onebigjson AS d LIMIT 1;
timeout is random as Marco Greco suggested

/opt/couchbase/bin/cbworkloadgen -n 127.0.0.1:8091 -u Administrator -p password -j -b onebigjson -i 1000 -s 2000000

Show

Ajay Bhullar added a comment - 06/Nov/20 2:03 PM we still see the error in 6.6.1-9177 tid: 0 loop: 22 {"elapsedTime": "1.489946ms", "executionTime": "1.300617ms", "resultSize": 0, "resultCount": 0, "errorCount": 2} [ {"msg": "Error performing bulk get operation - cause: read tcp 127.0.0.1:47996->127.0.0.1:11210: i/o timeout", "code": 12008, "retry": true} , {"msg": "Timeout 1ms exceeded", "code": 1080, "retry": true} ] Using sitarams script in the cbse and his repro steps: VM 8 cpu, 8gb machine single node cluster 2GB data node Bucket onebigjson 2GB Create primary index on onebigjson; SELECT META(d).id, d.* FROM onebigjson AS d LIMIT 1; timeout is random as Marco Greco suggested /opt/couchbase/bin/cbworkloadgen -n 127.0.0.1:8091 -u Administrator -p password -j -b onebigjson -i 1000 -s 2000000

Ajay Bhullar made changes - 06/Nov/20 2:03 PM

Resolution	Fixed [ 1 ]
Status	Resolved [ 5 ]	Reopened [ 4 ]

Marco Greco made changes - 06/Nov/20 2:10 PM

Resolution		Fixed [ 1 ]
Status	Reopened [ 4 ]	Resolved [ 5 ]

Hide

Permalink

Marco Greco added a comment - 06/Nov/20 2:11 PM

That is wholly correct. The testcase is designed to generate the timeouts.
What you mustn't see is the BAD MAGIC error messages following the timeouts, which in my testing you don't.

Show

Marco Greco added a comment - 06/Nov/20 2:11 PM That is wholly correct. The testcase is designed to generate the timeouts. What you mustn't see is the BAD MAGIC error messages following the timeouts, which in my testing you don't.

Hide

Permalink

Ajay Bhullar added a comment - 06/Nov/20 2:16 PM

based on marco's comments this is verified in 6.6.1-9177, not seeing any bad magic error messages, the bulk get operation error is always the one above, and happens infrequently. Ran script ~30 times

Show

Ajay Bhullar added a comment - 06/Nov/20 2:16 PM based on marco's comments this is verified in 6.6.1-9177, not seeing any bad magic error messages, the bulk get operation error is always the one above, and happens infrequently. Ran script ~30 times

Hide

Permalink

Ajay Bhullar added a comment - 06/Nov/20 2:43 PM

verified in 6.5.1-6312 using the repro steps above and based on marco's comments, we do not see bad magic error messages

Show

Ajay Bhullar added a comment - 06/Nov/20 2:43 PM verified in 6.5.1-6312 using the repro steps above and based on marco's comments, we do not see bad magic error messages

Ajay Bhullar made changes - 06/Nov/20 2:43 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

Wayne Siu made changes - 06/Nov/20 6:53 PM

Link

This issue blocks MB-42583 [ MB-42583 ]

Wayne Siu made changes - 01/Dec/20 7:29 PM

Fix Version/s

6.5.2 [ 17223 ]

Arunkumar Senthilnathan made changes - 17/Feb/21 12:24 PM

Labels

approved-for-6.6.1

approved-for-6.6.1 releasenote

Hide

Permalink

Couchbase Build Team added a comment - 09/Jun/21 12:22 PM

Build sync_gateway-3.0.0-223 contains gomemcached commit 8e4fcec with commit message:
~~MB-42555~~ mark connections as unusable on golang / os socket errors

Show

Couchbase Build Team added a comment - 09/Jun/21 12:22 PM Build sync_gateway-3.0.0-223 contains gomemcached commit 8e4fcec with commit message: MB-42555 mark connections as unusable on golang / os socket errors

Lynn Straus made changes - 17/Jun/21 3:27 PM

Fix Version/s

7.0.0 [ 17233 ]

Lynn Straus made changes - 17/Jun/21 3:32 PM

Fix Version/s

Cheshire-Cat [ 15915 ]

People

Assignee:: Mihir Kamdar

Reporter:: Marco Greco

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 06/Nov/20 2:39 AM

Updated:: 17/Jun/21 3:32 PM

Resolved:: 06/Nov/20 2:10 PM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-42555 mark connections as unusable on golang / os socket errors: Gerrit Review:

MB-42555 mark connections as unusable on golang / os socket errors: Gerrit Review:

PagerDuty

Couchbase Server

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty