Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
4.0.0, 4.1.2, 4.5.1, 4.6.4, 5.0.1, Cheshire-Cat, 6.0.2, 5.5.5, 6.6.1
-
Untriaged
-
1
-
Unknown
Description
Marking it as query, but it affects any component that uses go-couchbase or gomemcached.
When a gomemcached Get() ends prematurely (the most likely cause being a golang timeout),
client/transport.go:getResponse() leaves the response structure uninitialized, which means that it has a 0 status.
This is not picked as fatal, and the connection gets eventually reused, reading incorrectly, data left behind on the wire.
Attachments
| For Gerrit Dashboard: MB-42555 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 139570,1 | MB-42555 mark connections as unusable on golang / os socket errors | mad-hatter | gomemcached | Status: MERGED | +2 | +1 |
| 139715,2 | MB-42555 mark connections as unusable on golang / os socket errors | master | gomemcached | Status: MERGED | +2 | +1 |
Activity
| Field | Original Value | New Value |
|---|---|---|
| Business Impact/value | P3 - Nice to have [ 10553 ] | |
| Component/s | query [ 10718 ] | |
| Component/s | query [ 11829 ] | |
| Is this a Regression? | Unknown [ 10452 ] | |
| Key |
|
|
| Triage | Untriaged [ 10351 ] | |
| Affects Version/s | 5.5.5 [ 16166 ] | |
| Affects Version/s | 6.0.2 [ 15919 ] | |
| Affects Version/s | 5.0.1 [ 15052 ] | |
| Affects Version/s | 4.6.4 [ 14607 ] | |
| Affects Version/s | 4.5.1 [ 13411 ] | |
| Affects Version/s | 4.1.2 [ 13312 ] | |
| Affects Version/s | 4.0.0 [ 12148 ] | |
| Affects Version/s | Cheshire-Cat [ 15915 ] | |
| Affects Version/s | 6.6.1 [ 17002 ] | |
| Affects Version/s | 4.0.0 [ 12709 ] | |
| Affects Version/s | 4.1.2 [ 13611 ] | |
| Affects Version/s | 4.5.1 [ 13613 ] | |
| Affects Version/s | 5.0.1 [ 15071 ] | |
| Affects Version/s | 4.6.4 [ 15100 ] | |
| Affects Version/s | 5.5.5 [ 16245 ] | |
| Affects Version/s | 6.0.2 [ 16246 ] | |
| Affects Version/s | Cheshire-Cat [ 16249 ] | |
| Affects Version/s | 6.6.1 [ 17140 ] | |
| Workflow | CBSE workflow [ 184345 ] | classic default workflow [ 184346 ] |
| Project | Couchbase Support/Engineering Tasks [ 10061 ] | Couchbase Server [ 10010 ] |
| Link | This issue blocks CBSE-9129 [ CBSE-9129 ] |
| Fix Version/s | Cheshire-Cat [ 15915 ] | |
| Fix Version/s | 6.6.1 [ 17002 ] |
| Link | This issue blocks MB-40528 [ MB-40528 ] |
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Assignee | Marco Greco [ marco.greco ] | Mihir Kamdar [ mihir.kamdar ] |
| Labels | approved-for-6.6.1 |
| Link | This issue blocks MB-42576 [ MB-42576 ] |
Build couchbase-server-6.6.1-9177 contains gomemcached commit 7b3f110 with commit message:
MB-42555 mark connections as unusable on golang / os socket errors
verified in 7.0.0-3652 using sitarams script in the CBSE got a successful repro in 6.5.1 and in 7.0.0-3652 we dont see the bulk get error after running the script 10-15 times, running it twice triggered the issue in 6.5.1
Build couchbase-server-6.5.1-6312 contains gomemcached commit 7b3f110 with commit message:
MB-42555 mark connections as unusable on golang / os socket errors
we still see the error in 6.6.1-9177
tid: 0 loop: 22
{"elapsedTime": "1.489946ms", "executionTime": "1.300617ms", "resultSize": 0, "resultCount": 0, "errorCount": 2}[
{"msg": "Error performing bulk get operation - cause: read tcp 127.0.0.1:47996->127.0.0.1:11210: i/o timeout", "code": 12008, "retry": true},
{"msg": "Timeout 1ms exceeded", "code": 1080, "retry": true}]
Using sitarams script in the cbse and his repro steps:
VM 8 cpu, 8gb machine
single node cluster 2GB data node
Bucket onebigjson 2GB
Create primary index on onebigjson;
SELECT META(d).id, d.* FROM onebigjson AS d LIMIT 1;
timeout is random as Marco Greco suggested
/opt/couchbase/bin/cbworkloadgen -n 127.0.0.1:8091 -u Administrator -p password -j -b onebigjson -i 1000 -s 2000000
| Resolution | Fixed [ 1 ] | |
| Status | Resolved [ 5 ] | Reopened [ 4 ] |
| Resolution | Fixed [ 1 ] | |
| Status | Reopened [ 4 ] | Resolved [ 5 ] |
That is wholly correct. The testcase is designed to generate the timeouts.
What you mustn't see is the BAD MAGIC error messages following the timeouts, which in my testing you don't.
based on marco's comments this is verified in 6.6.1-9177, not seeing any bad magic error messages, the bulk get operation error is always the one above, and happens infrequently. Ran script ~30 times
verified in 6.5.1-6312 using the repro steps above and based on marco's comments, we do not see bad magic error messages
| Status | Resolved [ 5 ] | Closed [ 6 ] |
| Link | This issue blocks MB-42583 [ MB-42583 ] |
| Fix Version/s | 6.5.2 [ 17223 ] |
| Labels | approved-for-6.6.1 | approved-for-6.6.1 releasenote |
Build sync_gateway-3.0.0-223 contains gomemcached commit 8e4fcec with commit message:
MB-42555 mark connections as unusable on golang / os socket errors
| Fix Version/s | 7.0.0 [ 17233 ] |
| Fix Version/s | Cheshire-Cat [ 15915 ] |
Build couchbase-server-7.0.0-3652 contains gomemcached commit 8e4fcec with commit message:
MB-42555mark connections as unusable on golang / os socket errors