Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-4985-enterprise
-
Untriaged
-
Windows 64-bit
-
1
-
No
Description
Script to Repro
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_drop_recreate_rebalance.CollectionsDropRecreateRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=6,nodes_failover=2,bucket_spec=multi_bucket.buckets_1000_collections,recovery_type=delta,GROUP=set1'
|
Steps to Repro
1. Create a 5 node cluster
----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
----------------------------------------------------------------------
| 172.23.104.247 | kv | 7.0.0-4985-enterprise | 6.32833333333 | Cluster node |
| 172.23.105.1 | None | <--- IN — | ||
| 172.23.105.137 | None | <--- IN — | ||
| 172.23.105.183 | None | <--- IN — | ||
| 172.23.106.116 | None | <--- IN — |
----------------------------------------------------------------------
2. Create buckets/scopes/collections/data
2021-04-21 16:24:27,200 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
------------------------------------------------------------------------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
------------------------------------------------------------------------
| bucket1 | ephemeral | 2 | none | 0 | 5000 | 5242880000 | 502953456 | 170 |
| default | couchbase | 2 | none | 0 | 5000 | 5242880000 | 501996792 | 366227683 |
------------------------------------------------------------------------
3. Hard Failover 2 nodes
2021-04-21 16:24:27,236 | test | INFO | MainThread | [collections_drop_recreate_rebalance:load_collections_with_failover:234] failing over nodes [ip:172.23.104.247 port:8091 ssh_username:Administrator, ip:172.23.105.1 port:8091 ssh_username:Administrator]
|
They fail with following error
2021-04-21 16:27:00,142 | test | ERROR | pool-11-thread-25 | [rest_client:_http_request:748] POST http://172.23.104.247:8091/controller/failOver body: otpNode=ns_1%40172.23.105.1&allowUnsafe=false headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: status: 500, content: Cannot safely perform a failover at the moment Cannot safely perform a failover at the moment auth: Administrator:password
|
At this point we see the following error on 172.23.105.183
Administrator@WIN-1T98IIFH727 /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
|
$ grep CRITICAL *
|
memcached.log.000833.txt:2021-04-21T16:26:46.064372-07:00 CRITICAL 20572: RbacConfigReloadTask(): An error occurred while loading RBAC configuration from [c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached.rbac] [ {"ip":"127.0.0.1","port":59145} - {"ip":"127.0.0.1","port":11209} (System, <ud>@ns_server</ud>) ]: Failed to parse RBAC database: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
|
grep: rebalance: Is a directory
|
cbcollect_attached.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-