Description
None
Environment
java-client-3.1.2.jar
core-io-2.1.2.jar
Server 7.0.0-4678
Gerrit Reviews
None
Release Notes Description
None
Activity
Show:
Matt Ingenthron
updated the WorkflowDecember 22, 2022 at 6:46 AMCouchbase SDK Workflow
Couchbase SDK Workflow with Review
Michael Nitschinger
changed the StatusJune 8, 2021 at 9:48 AMIn Progress
Resolved
Michael Nitschinger
updated the ResolutionJune 8, 2021 at 9:48 AMNone
Fixed
Michael Nitschinger
changed the StatusJune 7, 2021 at 1:17 PMOpen
In Progress
Graham Pople
updated the Fix versionsJune 4, 2021 at 2:28 PMNone
2.2.0
Graham Pople
updated the Fix versionsJune 4, 2021 at 2:28 PM2.1.6
None
Michael Nitschinger
updated the Fix versionsMay 6, 2021 at 6:43 AM2.1.5
None
Michael Nitschinger
updated the Fix versionsMay 6, 2021 at 6:43 AMNone
2.1.6
Michael Nitschinger
updated the Fix versionsApril 7, 2021 at 1:29 PMNone
2.1.5
Michael Nitschinger
updated the Fix versionsApril 7, 2021 at 1:29 PM2.1.4
None
Graham Pople
changed the StatusMarch 22, 2021 at 4:13 PMNew
Open
Ashwin Govindarajulu
updated the DescriptionMarch 18, 2021 at 10:23 AM*Scenario*:
* Create Memcached bucket on 2 node cluster
* Open SDK client connection for create mc bucket
* Try to select non-existing scope/collection and perform document insert
*Observation*:
Insert op fails with AmibiguousTimeoutException followed by "Could not publish Event because the queue is full" messages. After this state, I could not re-select default_collection for an operation.
{noformat}
{'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":false,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TIMEOUT","requestId":14,"requestType":"InsertRequest","retried":19,"retryReasons":["ENDPOINT_NOT_WRITABLE","COLLECTION_MAP_REFRESH_IN_PROGRESS"],"service":{"bucket":"default","collection":"test","documentId":"test","opaque":"0x231","scope":"_default","type":"kv"},"timeoutMs":5000,"timings":{"encodingMicros":1153,"totalMicros":5012651}}', 'cas': 0, 'value': {}, 'key': 'test', 'status': False}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT3.089991955S, createdAt=125792097758670, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":69614,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["KV_COLLECTION_OUTDATED"],"service":{"bucket":"default","collection":"test","opaque":"0x11211","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997174,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.048678927S, createdAt=125792097919150, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":35585,"requestType":"GetCollectionIdRequest","retried":5,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x8d24","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997331,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356471665S, createdAt=125792097919833, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27192,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5b","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995833,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356771059S, createdAt=125792098233623, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27193,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5c","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995836,"serverMicros":0}}}
...
...
Could not publish Event because the queue is full. ServiceStateChangedEvent{severity=DEBUG, category=com.couchbase.service, duration=PT0S, createdAt=125792108984491, description=Service changed state from CONNECTED to DEGRADED, context=ServiceContext{bucket=default, coreId=0x8efb008700000001, remote=10.112.210.102:11210, type=KV}, cause=null}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT0.069118502S, createdAt=125792108923821, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastDispatchedTo":"10.112.210.102","reason":"SHUTDOWN","requestId":220637,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x36000","scope":"_default","type":"kv"},"timeoutMs":10000}}
Could not publish Event because the queue is full. RequestRetryScheduledEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0.5S, createdAt=125792108987382, description=Request GetCollectionIdRequest retry scheduled per RetryStrategy (Reason: KV_COLLECTION_OUTDATED), context=RequestContext{completed=false, coreId=0x8efb008700000001, idempotent=true, lastChannelId=8EFB008700000001/00000000548CFD8D, lastDispatchedFrom=10.112.210.1:55052, lastDispatchedTo=10.112.210.102:11210, requestId=124303, requestType=GetCollectionIdRequest, retried=4, retryReasons=[ENDPOINT_NOT_WRITABLE], service={bucket=default, collection=test, opaque=0x1e7b2, scope=_default, type=kv}, timeoutMs=10000, timings={dispatchMicros=2007922, serverMicros=0}}, cause=null}{noformat}
*Scenario*:
* Create Memcached bucket on 2 node cluster
* Open SDK client connection for create mc bucket
* Try to select non-existing scope/collection and perform document insert
*Observation*:
Insert op fails with AmibiguousTimeoutException followed by "Could not publish Event because the queue is full" messages. After this state, I see this queue-full message event during default_collection operations. (default-collection operation succeeds)
{noformat}{'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":false,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TIMEOUT","requestId":14,"requestType":"InsertRequest","retried":19,"retryReasons":["ENDPOINT_NOT_WRITABLE","COLLECTION_MAP_REFRESH_IN_PROGRESS"],"service":{"bucket":"default","collection":"test","documentId":"test","opaque":"0x231","scope":"_default","type":"kv"},"timeoutMs":5000,"timings":{"encodingMicros":1153,"totalMicros":5012651}}', 'cas': 0, 'value': {}, 'key': 'test', 'status': False}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT3.089991955S, createdAt=125792097758670, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":69614,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["KV_COLLECTION_OUTDATED"],"service":{"bucket":"default","collection":"test","opaque":"0x11211","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997174,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.048678927S, createdAt=125792097919150, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":35585,"requestType":"GetCollectionIdRequest","retried":5,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x8d24","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997331,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356471665S, createdAt=125792097919833, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27192,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5b","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995833,"serverMicros":0}}}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356771059S, createdAt=125792098233623, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27193,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5c","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995836,"serverMicros":0}}}
...
...
Could not publish Event because the queue is full. ServiceStateChangedEvent{severity=DEBUG, category=com.couchbase.service, duration=PT0S, createdAt=125792108984491, description=Service changed state from CONNECTED to DEGRADED, context=ServiceContext{bucket=default, coreId=0x8efb008700000001, remote=10.112.210.102:11210, type=KV}, cause=null}
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT0.069118502S, createdAt=125792108923821, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastDispatchedTo":"10.112.210.102","reason":"SHUTDOWN","requestId":220637,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x36000","scope":"_default","type":"kv"},"timeoutMs":10000}}
Could not publish Event because the queue is full. RequestRetryScheduledEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0.5S, createdAt=125792108987382, description=Request GetCollectionIdRequest retry scheduled per RetryStrategy (Reason: KV_COLLECTION_OUTDATED), context=RequestContext{completed=false, coreId=0x8efb008700000001, idempotent=true, lastChannelId=8EFB008700000001/00000000548CFD8D, lastDispatchedFrom=10.112.210.1:55052, lastDispatchedTo=10.112.210.102:11210, requestId=124303, requestType=GetCollectionIdRequest, retried=4, retryReasons=[ENDPOINT_NOT_WRITABLE], service={bucket=default, collection=test, opaque=0x1e7b2, scope=_default, type=kv}, timeoutMs=10000, timings={dispatchMicros=2007922, serverMicros=0}}, cause=null}{noformat}
Ashwin Govindarajulu
created the IssueMarch 18, 2021 at 10:00 AMSomething went wrong on our end
If this keeps happening, share this information with your admin, who should contact support.
Hash 5PN8S6
Trace 5c87562759ee4cf79c3ba81670e45f6e
Pinned fields
Click on the next to a field label to start pinning.
Instabug