Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40526

[Transaction] Seeing keys missing in doc_loading during swap_rebalance

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: User Error
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: test-execution
    • Environment:
      6.6.0-7883 Enterprise Edition
      couchbase-transactions-1.1.0-SNAPSHOT
      java-client-3.0.6-SNAPSHOT

      Description

      Build: 6.6.0-7883

      Scenario:

      1. 4 node cluster, 2 Couchbase bucket (replica=2)

        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
        | Node           | Services             | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_used  | Active / Replica  | Version               |
        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
        | 172.23.105.205 | kv                   | 54.025974026    | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324      | 6.6.0-7883-enterprise |
        | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534   | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838       | 6.6.0-7883-enterprise |
        | 172.23.105.206 | kv                   | 57.1428571429   | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584       | 6.6.0-7883-enterprise |
        | 172.23.105.159 | kv                   | 56.9553805774   | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680       | 6.6.0-7883-enterprise |
        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+

        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | Bucket   | Type    | Replicas | Durability | TTL | Items | RAM Quota  | RAM Used | Disk Used |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | bucket-0 | membase | 2        | none       | 0   | 9483  | 2986344448 | 93145992 | 439373036 |
        | bucket-1 | membase | 2        | none       | 0   | 10229 | 2986344448 | 97605584 | 540662594 |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        

      1. Load few docs into the buckets using transaction
      2. Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
      3. Start parallel doc_loading using transaction during rebalance phase and validate

      Observation:

      At the end of the rebalance during data validation seeing keys missing from the bucket.

      Pcap files https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip

      Test case to run:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i nodes.ini -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True -p rerun=False,get-cbcollect-info=False -m rest'

        Attachments

        1. key_missing.test
          144 kB
        2. KeyMissing Wireshark.png
          KeyMissing Wireshark.png
          218 kB
        3. test.log
          43 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            ashwin.govindarajulu Ashwin Govindarajulu
            Reporter:
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Due:
              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty