Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Goldfish GA
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 42, Analytics Sprint 43
Description
- Create a 3 node provisioned cluster, a magma bucket, 10 collections and load 100M items in each with 1B items in total.
- Create columnar instance with 2 nodes.
- Create 10 remote collection each ingesting 100M items.
- Ingestion completed.
- Start an upsert kv workload on provisioned cluster.
- Start query workload.
- Scale Columnar from 2 -> 4 -> 8 nodes. All successful.
- Scale down from 8 to 4 nodes
- EjectNodes = ['ns_1@svc-da-node-004.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com']. Done in less than 60s
- EjectNodes = ['ns_1@svc-da-node-001.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com']. Done in less than 60s
- EjectNodes = ['ns_1@svc-da-node-007.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com']. Done in less than 2m but cluster balance state is FALSE
- Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-003.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-005.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-006.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-008.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 2e0b51811d1a8f23e27034aa1d45ed69 === Completed in <60s but state is still FALSE - Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-003.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-005.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-006.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-008.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b03342d6121755eaeedd6c19d3fa1bcf === Done in 20s but state is still FALSE - Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-003.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-005.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-006.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-008.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 66529572d8002e2f136f2afe3edcf689 === Completed in 1 hour BUT state is still FALSE - Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-003.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-005.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-006.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com',
'ns_1@svc-da-node-008.oglyir3dvb8ndilj.pl.nonprod-project-avengers.com'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 486691285a4cc8773ecf74ab6f7328aa === Rebalance is hung since 2 hours...
Cluster state:
"balanced": false, |
"servicesNeedRebalance": [ |
{
|
"code": "service_not_balanced", |
"description": "Service needs rebalance.", |
"services": [ |
"cbas"
|
]
|
}
|
]
|
QE Test |
sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/couchbase_capella_volume_3_new_dummy.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.goldfish.GoldfishVolume.Columnar.test_rebalance,num_items=100000000,num_buckets=1,bucket_names=GleamBook,bucket_type=membase,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,maxttl=10,pc=20,gsi_nodes=3,cbas_nodes=3,fts_nodes=3,kv_nodes=3,n1ql_nodes=2,kv_disk=1000,n1ql_disk=50,gsi_disk=500,fts_disk=1000,cbas_disk=1000,kv_compute=c5.4xlarge,gsi_compute=c5.4xlarge,n1ql_compute=c5.4xlarge,fts_compute=c5.4xlarge,cbas_compute=c5.4xlarge,mutation_perc=20,key_type=CircularKey,capella_run=true,services=data,rebl_services=,max_rebl_nodes=27,provider=AWS,region=us-east-1,type=GP3,size=1000,ops_rate=100000,skip_teardown_cleanup=false,wait_timeout=14400,index_timeout=28800,runtype=columnar1,skip_init=false,rebl_ops_rate=10000,collections=10,valType=Hotel,expiry=true,v_scaling=true,h_scaling=true,horizontal_scale=1,clients_per_db=10,track_failures=False,onPremMongo=False,num_clusters=1 -m rest'
|
Want to summarize the issues noticed:
- Why is the cluster balance state is False after the 6th node is properly removed from the cluster during 8->4 scaling.
- While CP trigger a noop with 5 nodes in cluster why is rebalance taking an hour while previous rebalance out was done in few seconds? What is analytics doing during a noop rebalance? noop means no change cluster topology.
- Why is the last 5 nodes noop rebalance operation hung?
Attachments
Issue Links
- relates to
-
MB-61999 On rebalance completion, ns_server clears rebalance indicator before refreshing isBalanced checks from services
- Open