Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
5.5.0-2211
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Steps to Repro
./testrunner -i /tmp/testexec.5938.ini get-cbcollect-info=True,GROUP=rebalance_multi,use_memory_manager=False -t eventing.eventing_rebalance.EventingRebalance.test_kv_eventing_rebalance_with_multiple_functions_deployed,nodes_init=7,services_init=kv-kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=1,services_in=eventing,server_out=4,GROUP=rebalance_multi
|
./testrunner -i /tmp/testexec.5938.ini get-cbcollect-info=True,GROUP=rebalance_multi,use_memory_manager=False -t eventing.eventing_rebalance.EventingRebalance.test_kv_eventing_rebalance_with_multiple_functions_deployed,nodes_init=7,services_init=kv-kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=1,services_in=kv,server_out=4,GROUP=rebalance_multi
|
Handler code has randomization logic as shown below
function OnUpdate(doc, meta) {
|
log('document', doc);
|
try {
|
var time_rand = random_gen();
|
dst_bucket[meta.id + time_rand] = doc;
|
} catch(e) {
|
var time_rand = random_gen();
|
dst_bucket[meta.id + time_rand] = doc;
|
}
|
}
|
function random_gen(){
|
var rand = Math.floor(Math.random() * 20000000) * Math.floor(Math.random() * 20000000);
|
var time_rand = Math.round((new Date()).getTime() / 1000) + rand;
|
return time_rand;
|
}
|
See the table below for expected and actual docs in destination bucket.
| Test | Expected Docs | Actual Docs | % increase duplicate mutations |
|---|---|---|---|
| Eventing Swap rebalance | 20160 | 33610 | 66% |
| KV Swap rebalance | 20160 | 37070 | 84% |
I understand duplicate docs are expected during rebalance because of our rebalance retry logic. I had a discussion with Abhishek Singh regarding this. Even though reducing retries or improving performance of rebalance might not be high priority at this point in time I think we should have a way to track number of duplicate mutations we created during rebalance. 84% increase in mutations during kv rebalance seems pretty high and I assume this would play a big role in how long it takes to complete a kv rebalance when eventing is involved because of the increased load. I also think this would play a big role in sizing of a eventing node (If we have plans to consider this into the equation)
Attaching logs.
Please feel free to move it to a Task if you think it is one.