Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.1.2
-
java-client-3.1.2.jar
core-io-2.1.2.jar
Server 7.0.0-4678
-
1
Description
Scenario:
- Create Memcached bucket on 2 node cluster
- Open SDK client connection for create mc bucket
- Try to select non-existing scope/collection and perform document insert
Observation:
Insert op fails with AmibiguousTimeoutException followed by "Could not publish Event because the queue is full" messages. After this state, I see this queue-full message event during default_collection operations. (default-collection operation succeeds)
{'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":false,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TIMEOUT","requestId":14,"requestType":"InsertRequest","retried":19,"retryReasons":["ENDPOINT_NOT_WRITABLE","COLLECTION_MAP_REFRESH_IN_PROGRESS"],"service":{"bucket":"default","collection":"test","documentId":"test","opaque":"0x231","scope":"_default","type":"kv"},"timeoutMs":5000,"timings":{"encodingMicros":1153,"totalMicros":5012651}}', 'cas': 0, 'value': {}, 'key': 'test', 'status': False}
|
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT3.089991955S, createdAt=125792097758670, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":69614,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["KV_COLLECTION_OUTDATED"],"service":{"bucket":"default","collection":"test","opaque":"0x11211","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997174,"serverMicros":0}}}
|
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.048678927S, createdAt=125792097919150, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: TOO_MANY_REQUESTS_IN_RETRY {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"TOO_MANY_REQUESTS_IN_RETRY","requestId":35585,"requestType":"GetCollectionIdRequest","retried":5,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x8d24","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":1997331,"serverMicros":0}}}
|
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356471665S, createdAt=125792097919833, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27192,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5b","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995833,"serverMicros":0}}}
|
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT4.356771059S, createdAt=125792098233623, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastChannelId":"8EFB008700000001/00000000548CFD8D","lastDispatchedFrom":"10.112.210.1:55052","lastDispatchedTo":"10.112.210.102:11210","reason":"SHUTDOWN","requestId":27193,"requestType":"GetCollectionIdRequest","retried":9,"retryReasons":["KV_COLLECTION_OUTDATED","ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x6c5c","scope":"_default","type":"kv"},"timeoutMs":10000,"timings":{"dispatchMicros":995836,"serverMicros":0}}}
|
...
|
...
|
Could not publish Event because the queue is full. ServiceStateChangedEvent{severity=DEBUG, category=com.couchbase.service, duration=PT0S, createdAt=125792108984491, description=Service changed state from CONNECTED to DEGRADED, context=ServiceContext{bucket=default, coreId=0x8efb008700000001, remote=10.112.210.102:11210, type=KV}, cause=null}
|
Could not publish Event because the queue is full. CollectionMapRefreshFailedEvent{severity=WARN, category=com.couchbase.config, duration=PT0.069118502S, createdAt=125792108923821, description=Collection Map refresh (CollectionIdentifier{bucket='default', scope=Optional[_default], collection=Optional[test], isDefault=false}) failed: FAILED, context=CoreContext{coreId=0x8efb008700000001}, cause=com.couchbase.client.core.error.RequestCanceledException: GetCollectionIdRequest, Reason: SHUTDOWN {"cancelled":true,"completed":true,"coreId":"0x8efb008700000001","idempotent":true,"lastDispatchedTo":"10.112.210.102","reason":"SHUTDOWN","requestId":220637,"requestType":"GetCollectionIdRequest","retried":2,"retryReasons":["ENDPOINT_NOT_WRITABLE"],"service":{"bucket":"default","collection":"test","opaque":"0x36000","scope":"_default","type":"kv"},"timeoutMs":10000}}
|
Could not publish Event because the queue is full. RequestRetryScheduledEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0.5S, createdAt=125792108987382, description=Request GetCollectionIdRequest retry scheduled per RetryStrategy (Reason: KV_COLLECTION_OUTDATED), context=RequestContext{completed=false, coreId=0x8efb008700000001, idempotent=true, lastChannelId=8EFB008700000001/00000000548CFD8D, lastDispatchedFrom=10.112.210.1:55052, lastDispatchedTo=10.112.210.102:11210, requestId=124303, requestType=GetCollectionIdRequest, retried=4, retryReasons=[ENDPOINT_NOT_WRITABLE], service={bucket=default, collection=test, opaque=0x1e7b2, scope=_default, type=kv}, timeoutMs=10000, timings={dispatchMicros=2007922, serverMicros=0}}, cause=null}
|