Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
Cheshire-Cat
-
Centos 7 64-bit; Couchbase Enterprise build 7.0.0-2637
-
Untriaged
-
Centos 64-bit
-
-
1
-
No
Description
Summary:
Some Scope/collection creation/drop fails with
"error: 503 reason: unknown "Operation is unsafe at this time. Retry later."
Unsure if this is something expected as with some sleep put (or retries later) to ensure these drops/creation happen after rebalance has started progress, it doesn't throw this error.
Steps to Reproduce
- Create a 3 node cluster
2020-07-28 06:01:59,322 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------++-------------Nodes Services Status
-----------------------++-------------
172.23.105.211 | kv | Cluster node |
172.23.105.212 | None | <--- IN — |
172.23.105.213 | None | <--- IN — |
-----------------------++-------------
2. Create buckets with collections and load data
2020-07-28 06:12:05,461 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-----------------+++-------------------------------+---------------------------------
Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-----------------+++-------------------------------+---------------------------------
bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 184798896 | 281126164 |
bucket2 | ephemeral | 3 | none | 0 | 3000 | 629145600 | 266479728 | 102 |
default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 511031552 | 455611628 |
-----------------+++-------------------------------+---------------------------------
3. Start rebalance-in op with CRUD on collections in parallel
Some CRUD on collections fail. Taking one such example:
2020-07-28 06:12:08,174 | infra | ERROR | Thread-1839 | [Rest_Connection:_http_request:205] DELETE http://172.23.105.211:8091/pools/default/buckets/bucket2/collections/_default/mA body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 503 reason: unknown "Operation is unsafe at this time. Retry later." auth: Administrator:password |
2020-07-28 06:12:08,181 | test | ERROR | Thread-1839 | [bucket_ready_functions:drop_collection:814] Collection 'bucket2:_default:mA' delete failed: "Operation is unsafe at this time. Retry later." |
Rebalance starts progress at:
2020-07-28 06:12:12,269 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
Nodes | Services | Status |
-----------------------++-------------
172.23.105.212 | kv | Cluster node |
172.23.105.213 | kv | Cluster node |
172.23.105.211 | kv | Cluster node |
172.23.105.215 | None | <--- IN — |
172.23.105.217 | None | <--- IN — |
-----------------------++-------------
Have attached the consoleText. Screenshot of UI showing collection mA still present. Have attached the cb-collect-info.