Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Enterprise Edition 7.1.0 build 2149
-
Untriaged
-
1
-
Yes
-
KV 2022-Jan
Description
Steps:
1. Create a 4 KV node and 2 index/n1ql node cluster
2. Create a magma default bucket. Create 50 collections.
3. Load 20M items per collections that makes it 1B in the bucket. Until here the items in Active and Replica vB are all maching.
4. Upsert all 1B items. While upserting 109,132 items got into AmbiguousTimeoutException
5. After the load finished, retry upserts are all successful.
6. Wait for the persistence to finish.
7. Validate the items count across active/replica vB
8. Total items in Active=1000,000,000 while in replica here are the details:
vb_replica_curr_items: 1000014165(Actual) != 1000000000(Expected)
172.23.107.240: 249860750
172.23.107.76: 250114000
172.23.107.221: 249938900
172.23.107.232: 250100515
Cluster: http://172.23.107.221:8091/ui/index.html
I have observed that there is an items count mismatch between active/replica cbstats. Ran cbstats to cross check the numbers and found the kvstores where there is a mismatch. Here they are:
172.23.107.232 kvstore 67 count==978846
|
172.23.107.221 kvstore 67 count==975350
|
------------------------------------------------------
|
172.23.107.232 kvstore 73 count==979273
|
172.23.107.221 kvstore 73 count==975700
|
------------------------------------------------------
|
172.23.107.232 kvstore 78 count==985236
|
172.23.107.221 kvstore 78 count==981700
|
------------------------------------------------------
|
172.23.107.232 kvstore 80 count==970310
|
172.23.107.221 kvstore 80 count==966750
|
978846+979273+985236+970310-975350-975700-981700-966750 = 14165 and thats the actual total diff in the stats.