Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
6.6.2-9588 -> 7.0.0-5141
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Scripts to Repro
1. Run the 6.6.2 longevity test for 3 days.
./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. It had 27 nodes at the end of the test.
3. Added 6 7.0.0(172.23.105.102,172.23.105.62,172.23.106.232,172.23.106.239,172.23.106.37, 172.23.106.246) nodes and rebalanced in and removed 6 node from 6.6.2(172.23.110.75,172.23.110.76,172.23.105.61,172.23.106.191,172.23.106.209,172.23.106.70)
and rebalanced out.
4. Failed over 6 nodes and graceful failover + recovery + rebalance.
5. Now swap rebalance 6 nodes. 2 data + 2 index + 1 eventing + 1 analytics.
6. Then again did bunch of rebalances(stop/start and rebalance out eventing) to workaround MB-46274 and MB-46246.
7. Started a 5 node (2 indexing + 1 data + 1 eventing + 1 analytics) graceful failover + delta recovery + rebalance that has hung at indexing service for 6+ hours now.
Rebalance
Starting rebalance, KeepNodes = ['ns_1@172.23.104.15','ns_1@172.23.104.214',
|
'ns_1@172.23.104.232','ns_1@172.23.104.244',
|
'ns_1@172.23.104.245','ns_1@172.23.105.102',
|
'ns_1@172.23.105.109','ns_1@172.23.105.112',
|
'ns_1@172.23.105.118','ns_1@172.23.105.164',
|
'ns_1@172.23.105.206','ns_1@172.23.105.210',
|
'ns_1@172.23.105.25','ns_1@172.23.105.61',
|
'ns_1@172.23.105.62','ns_1@172.23.105.86',
|
'ns_1@172.23.105.90','ns_1@172.23.105.93',
|
'ns_1@172.23.106.117','ns_1@172.23.106.191',
|
'ns_1@172.23.106.207','ns_1@172.23.106.209',
|
'ns_1@172.23.106.225','ns_1@172.23.106.232',
|
'ns_1@172.23.106.239','ns_1@172.23.106.246',
|
'ns_1@172.23.106.32','ns_1@172.23.106.37',
|
'ns_1@172.23.106.54','ns_1@172.23.110.75',
|
'ns_1@172.23.110.76'], EjectNodes = [], Failed over and being ejected nodes = [], Delta recovery nodes = ['ns_1@172.23.105.86'], Delta recovery buckets = all; Operation Id = 0cff600429c7b845ea52cac885d66526
|
Rebalance Status
[root@localhost ~]# date ;/opt/couchbase/bin/couchbase-cli rebalance-status -c 172.23.105.102 --username Administrator --password password
|
Thu May 13 04:45:58 PDT 2021
|
{
|
"status": "running",
|
"msg": "Rebalance is running",
|
"details": {
|
"progress": 57.87096774193547,
|
"refresh": 0.25,
|
"totalBuckets": 10,
|
"curBucket": 10,
|
"curBucketName": "default",
|
"docsRemaining": 0
|
}
|
}
|
[root@localhost ~]#
|
[root@localhost ~]# date ;/opt/couchbase/bin/couchbase-cli rebalance-status -c 172.23.105.102 --username Administrator --password password
|
Thu May 13 10:24:13 PDT 2021
|
{
|
"status": "running",
|
"msg": "Rebalance is running",
|
"details": {
|
"progress": 57.87096774193547,
|
"refresh": 0.25,
|
"totalBuckets": 10,
|
"curBucket": 10,
|
"curBucketName": "default",
|
"docsRemaining": 0
|
}
|
}
|
[root@localhost ~]#
|
cbcollect_info attached. Not sure if its dup of MB-46274.
Attachments
Issue Links
- relates to
-
-
- Closed
-