Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47952

Item seems to be deleted even when there were no delete ops performed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • couchbase-bucket
    • Centos 7 64 bit; CB EE 7.1.0-1132

    Description

      Steps
      1. Create a 6 node kv cluster

      2021-08-12 00:23:49,779 | test  | INFO    | pool-3-thread-9 | [table_view:display:72] Rebalance Overview
      +----------------+----------+-----------------------+---------------+--------------+
      | Nodes          | Services | Version               | CPU           | Status       |
      +----------------+----------+-----------------------+---------------+--------------+
      | 172.23.105.175 | kv       | 7.1.0-1132-enterprise | 1.99448068239 | Cluster node |
      | 172.23.106.233 | None     |                       |               | <--- IN ---  |
      | 172.23.106.236 | None     |                       |               | <--- IN ---  |
      | 172.23.106.238 | None     |                       |               | <--- IN ---  |
      | 172.23.106.250 | None     |                       |               | <--- IN ---  |
      | 172.23.106.251 | None     |                       |               | <--- IN ---  |
      +----------------+----------+-----------------------+---------------+--------------+

      2. Create a magma bucket with full eviction policy, with 1 replica, 50 scopes, and 100 collections in each scope

      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+
      | Bucket  | Type      | Storage Backend | Replicas | Durability | TTL | Items    | RAM Quota | RAM Used | Disk Used | ARR |
      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+
      | default | couchbase | magma           | 1        | none       | 0   | 12838277 | 58.59 GiB | 4.99 GiB | 0.0 Byte  | 100 |
      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+----------+-----------+-----+

      3. Load 80M docs in each collection (8 billion items in total across 1000 collections). This took ~ 25 hours

      2021-08-12 00:24:27,944 | test  | INFO    | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4849] Performing scope/collection specific operations

      4. Validate the items count

      2021-08-13 02:20:54,482 | test  | INFO    | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4603] Validating collection stats and item counts

      Here, total item count was 1 less than 8 billion

      Using cbstats figured out scope, collection the item was missing.
      Scope=rjaEYMvvkjD0O-0-466000;
      collection=a2c2eOJLn94mACh-0-471000
      (I performed the first set of logs collection here. Use the logs link of the ticket to access this set of logs)
      (After collecting first set of logs, I updated the bucket replica to 2, and performed some failover and recovery rebalance operations over the weekend, but did not perform any mutations to the documents) 

      Yesterday, I attempted to read all docs from the above scope, collection as suggested by Dave Rigby to figure out the item that was missing was:
      test_collections-4895787

      (Performed another set of logs collection here. Refer 
       http://supportal.couchbase.com/snapshot/ba0d816c9cd6ad5458820c542015c602::1
      )

      Thanks to Ashwin Govindarajulu as he made an observation that this item had been marked as deleted. He notes his observation below:
      (Ashwin will update here shortly)

      Attachments

        Issue Links

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              sumedh.basarkod Sumedh Basarkod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty