Data mismatch between source and destination buckets for Eventing multi function + rebalance tests

QE Test

./testrunner -i /data/workspace/debian-p0-eventing-vset00-00-multihandler_multicollection_rebalance_set3/testexec.22845.ini -p get-cbcollect-info=True,GROUP=multihandler_multicollection_rebalance_set3,java_sdk_client=True,global_function_scope=True,bucket_size=100,default_bucket=False,default_bucket=False,sirius_url=http://172.23.120.103:4000 -t eventing.eventing_multihandler.EventingMultiHandler.test_multiple_handle_multiple_collections_rebalance_in,nodes_init=2,services_init=kv-eventing,dataset=default,groups=simple,reset_services=True,skip_cleanup=True,num_src_buckets=2,num_dst_buckets=5,num_handlers=10,deploy_handler=10,worker_count=1,handler_code=handler_code/ABO/insert_template.js,sequential=False,num_pause=1,GROUP=multihandler_multicollection_rebalance_set3,global_function_scope=True

Test Steps

1. Cluster consists of 1 KV and Eventing node each.
2. Create following buckets and required set of scopes and collections required as part of the test - src_bucket_0, src_bucket_0, dst_bucket_0, dst_bucket_1, dst_bucket_2, dst_bucket_3, dst_bucket_4*and *metadata bucket for storing Eventing metadata.
3. Create 10 Eventing functions.
4. In each of the 10 functions, we are listening for changes in 1 of the collections present in src_bucket and writing the corresponding document in 1 of the 5 destination buckets.
5. Sample Eventing handler code -

   function OnUpdate(doc, meta) {

   log("Doc created/updated", meta.id);

   try{

   var id={"id":'func_0'+'_'+meta.id}

   var result1= couchbase.insert(dst_bucket,id,doc);

   log(result1);

   }catch(e){

   log("error:",e);

   }

   }

   function OnDelete(meta, options) {

   log("Doc deleted/expired", meta.id);

   var doc={"id":'func_0'+'_'+meta.id}

   var result = couchbase.delete(dst_bucket,doc);

   log(result);

   }

6. Load 2016 docs in all source buckets.
7. Deploy all 5 functions.
8. While mutations are being processed, rebalance in 2nd Eventing node.
9. Verify document count matches with the expected value for all 5 destination buckets.
10. For 1 of the destination buckets, there is a mismatch of between expected number of documents and actual document count.

    [2023-08-28 07:03:49,381] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:172.23.106.21 port:8091 ssh_username:root]

    [2023-08-28 07:03:49,383] - [remote_util:302] INFO - SSH Connecting to 172.23.106.21 with username:root, attempt#1 of 5

    [2023-08-28 07:03:49,503] - [remote_util:338] INFO - SSH Connected to 172.23.106.21 as root

    [2023-08-28 07:03:49,637] - [remote_util:3475] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True

    [2023-08-28 07:03:49,904] - [remote_util:3634] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11

    [2023-08-28 07:03:50,057] - [basetestcase:833] INFO - sleep for 30 secs. Waiting for mutation processing to complete, Current count : 3025 Expected count : 4032 ...

Note

This particular job was passing till 7.2.1-5922 build but is failing for all builds from 5923 onwards. I have run the tests multiple times on preceding and succeeding builds to rule out any infra issues.

Following is the only change which went into 5923 build -
http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.2.1&fromBuild=5922&toVersion=7.2.1&toBuild=5923&f_kv_engine=on