curr_items is greater than the actual number of items and fluctuates when there are ongoing mutations

Build 4.6.5-4742.

Setup:

• 4 nodes
• 1 bucket, 1 replica, full eviction

Step 1 -
The test loads 1B items into the bucket. All insert operations succeeded.

Steps 2 -
The test reads a sub-set of the data (5%).

Steps 3 -
The test starts a workload (90% read operations, 10% update operations, 15K total ops/sec).

At this point multiple curr_items counters demonstrate at least three problems:

• Aggregated curr_items > 1B
• Aggregated curr_items slightly changes over time
• MB-29816

Here are some initial samples (curr_items):

1000000138

1000000232

1000000313

1000000197

1000000261

1000000277

1000000227

1000000196

1000000341

1000000246

1000000170

1000000342

1000000318

1000000217

1000000234

1000000443

1000000174

1000000166

1000000346

1000000298

and vb_replica_curr_items

1000001113

1000001958

1000002424

1000002058

1000003031

1000002787

1000002123

1000002375

1000002708

1000001901

1000001944

1000002711

1000003785

1000001980

1000002119

1000003554

1000001858

1000002373

1000002982

1000002838

Step 4 -
The test swaps one of the nodes (172.23.96.103 -> 172.23.96.104) 30 minutes later. Rebalance successfully finishes after ~1 hour. All three issues still can be observed.

Step 5 -
The test stops the ongoing workload. curr_items counters still exceed 1B but no longer fluctuate. vb_replica_curr_items > vb_active_curr_items.

Step 6 -
The test restarts the entire cluster by rebooting every server. curr_items is precisely 1B after warmup, vb_replica_curr_items == vb_active_curr_items.