Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
7.2.0-5285
-
Untriaged
-
-
0
-
Unknown
Description
Steps:
- Create a 2 KV and 1 index/query node cluster.
- Create a magma bucket(replicas=1) and collections(total collection count including default collections is 51)
- Create 500000000 items sequentially(After creation of few thousands of documents update
- Update 500000000 created in above step
- Create 500000000 items sequentially
- Update 500000000 created in above step
- Create five indexes Wait for index building.
- Rebalance in KV with Loading of docs. (Rebalance completed successfully)
- Rebalance Out KV with Loading of docs.(Rebalance completed successfully)
- Rebalance In_Out KV with Loading of docs.
- Pause the rebalance and Enable CDC bucket_history_retention_seconds=259200,bucket_history_retention_bytes=10000000000000)
- Again trigger rebalance in_out KV with loading of docs (Rebalance completed successfully)
- Gracefull failover a node , Add a node and trigger rebalance(A swap rebalance)
- Rebalance exited with reason {service_rebalance_failed,index,
{agent_died,<34340.5783.0>,
Note:
Cluster is still in same state
http://172.23.110.64:8091/ui/index.html#/servers/list?commonBucket=GleamBookUsers0&scenarioZoom=minute&scenario=6fitqfa3g
This looks similar to MB-56170, But since it failed at a different step, Logging a new bug
Error:
Rebalance exited with reason {service_rebalance_failed,index,
|
{agent_died,<34340.5783.0>,
|
{linked_process_died,<34340.2799.253>,
|
{'ns_1@172.23.110.67',
|
{timeout,
|
{gen_server,call,
|
[<34340.5835.0>,
|
{call,"ServiceAPI.StartTopologyChange",
|
#Fun<json_rpc_connection.0.86436583>,
|
#{timeout => 60000}},
|
60000]}}}}}}.
|
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=2,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=error,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey -m rest'
|