Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50620

14165 items more in replica than active vBuckets as per cbstats.

    XMLWordPrintable

Details

    Description

      Steps:
      1. Create a 4 KV node and 2 index/n1ql node cluster
      2. Create a magma default bucket. Create 50 collections.
      3. Load 20M items per collections that makes it 1B in the bucket. Until here the items in Active and Replica vB are all maching.
      4. Upsert all 1B items. While upserting 109,132 items got into AmbiguousTimeoutException
      5. After the load finished, retry upserts are all successful.
      6. Wait for the persistence to finish.
      7. Validate the items count across active/replica vB
      8. Total items in Active=1000,000,000 while in replica here are the details:

      vb_replica_curr_items: 1000014165(Actual) != 1000000000(Expected)
      172.23.107.240: 249860750
      172.23.107.76: 250114000
      172.23.107.221: 249938900
      172.23.107.232: 250100515

      Cluster: http://172.23.107.221:8091/ui/index.html

      I have observed that there is an items count mismatch between active/replica cbstats. Ran cbstats to cross check the numbers and found the kvstores where there is a mismatch. Here they are:

      172.23.107.232	kvstore	67	count==978846
      172.23.107.221	kvstore	67	count==975350
      ------------------------------------------------------
      172.23.107.232	kvstore	73	count==979273
      172.23.107.221	kvstore	73	count==975700
      ------------------------------------------------------
      172.23.107.232	kvstore	78	count==985236
      172.23.107.221	kvstore	78	count==981700
      ------------------------------------------------------
      172.23.107.232	kvstore	80	count==970310
      172.23.107.221	kvstore	80	count==966750
      

      978846+979273+985236+970310-975350-975700-981700-966750 = 14165 and thats the actual total diff in the stats.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty