Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
7.1.0-1312
-
Untriaged
-
-
1
-
Unknown
-
KV 2021-Nov, KV 2021-Dec
Description
1. Create a 3 kv node and 1 index/n1ql node cluster
2. Create magma bucket, 100 collections under default scope
3. Load 5B items and upsert them
4. Load another 5B items and upsert them as well
5. Create 100 indexes on 100 collections and build them. Start 100 QPS
6. Rebalance In 1 KV node. Rebalance failed
7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm
Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM
Total Disk Writes are varying between 5-30Mbps
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'
|
