Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Centos 7 64 bit; CB EE 7.1.0-1132
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Steps
1. Create a 6 node kv cluster
2021-08-12 00:23:49,779 | test | INFO | pool-3-thread-9 | [table_view:display:72] Rebalance Overview
|
+----------------+----------+-----------------------+---------------+--------------+
|
| Nodes | Services | Version | CPU | Status |
|
+----------------+----------+-----------------------+---------------+--------------+
|
| 172.23.105.175 | kv | 7.1.0-1132-enterprise | 1.99448068239 | Cluster node |
|
| 172.23.106.233 | None | | | <--- IN --- |
|
| 172.23.106.236 | None | | | <--- IN --- |
|
| 172.23.106.238 | None | | | <--- IN --- |
|
| 172.23.106.250 | None | | | <--- IN --- |
|
| 172.23.106.251 | None | | | <--- IN --- |
|
+----------------+----------+-----------------------+---------------+--------------+
|
2. Create a magma bucket with full eviction policy, with 1 replica, 50 scopes, and 100 collections in each scope
+---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+
|
| Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR |
|
+---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+
|
| default | couchbase | magma | 1 | none | 0 | 12838277 | 58.59 GiB | 4.99 GiB | 0.0 Byte | 100 |
|
+---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+
|
3. Load 80M docs in each collection (8 billion items in total across 1000 collections). This took ~ 25 hours
2021-08-12 00:24:27,944 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4849] Performing scope/collection specific operations
|
4. Validate the items count
2021-08-13 02:20:54,482 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4603] Validating collection stats and item counts
|
Here, total item count was 1 less than 8 billion
Using cbstats figured out scope, collection the item was missing.
Scope=rjaEYMvvkjD0O-0-466000;
collection=a2c2eOJLn94mACh-0-471000
(I performed the first set of logs collection here. Use the logs link of the ticket to access this set of logs)
(After collecting first set of logs, I updated the bucket replica to 2, and performed some failover and recovery rebalance operations over the weekend, but did not perform any mutations to the documents)
Yesterday, I attempted to read all docs from the above scope, collection as suggested by Dave Rigby to figure out the item that was missing was:
test_collections-4895787
(Performed another set of logs collection here. Refer
http://supportal.couchbase.com/snapshot/ba0d816c9cd6ad5458820c542015c602::1
)
Thanks to Ashwin Govindarajulu as he made an observation that this item had been marked as deleted. He notes his observation below:
(Ashwin will update here shortly)
