Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40526

[Transaction] Seeing keys missing in doc_loading during swap_rebalance

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: User Error
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: test-execution
    • Environment:
      6.6.0-7883 Enterprise Edition
      couchbase-transactions-1.1.0-SNAPSHOT
      java-client-3.0.6-SNAPSHOT

      Description

      Build: 6.6.0-7883

      Scenario:

      1. 4 node cluster, 2 Couchbase bucket (replica=2)

        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
        | Node           | Services             | CPU_utilization | Mem_total  | Mem_free   | Swap_mem_used  | Active / Replica  | Version               |
        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
        | 172.23.105.205 | kv                   | 54.025974026    | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324      | 6.6.0-7883-enterprise |
        | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534   | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838       | 6.6.0-7883-enterprise |
        | 172.23.105.206 | kv                   | 57.1428571429   | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584       | 6.6.0-7883-enterprise |
        | 172.23.105.159 | kv                   | 56.9553805774   | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680       | 6.6.0-7883-enterprise |
        +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+

        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | Bucket   | Type    | Replicas | Durability | TTL | Items | RAM Quota  | RAM Used | Disk Used |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        | bucket-0 | membase | 2        | none       | 0   | 9483  | 2986344448 | 93145992 | 439373036 |
        | bucket-1 | membase | 2        | none       | 0   | 10229 | 2986344448 | 97605584 | 540662594 |
        +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
        

      1. Load few docs into the buckets using transaction
      2. Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
      3. Start parallel doc_loading using transaction during rebalance phase and validate

      Observation:

      At the end of the rebalance during data validation seeing keys missing from the bucket.

      Pcap files https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip

      Test case to run:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i nodes.ini -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True -p rerun=False,get-cbcollect-info=False -m rest'

        Attachments

        1. key_missing.test
          144 kB
        2. KeyMissing Wireshark.png
          KeyMissing Wireshark.png
          218 kB
        3. test.log
          43 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          ashwin.govindarajulu Ashwin Govindarajulu created issue -
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Field Original Value New Value
          Description *Build*: 6.6.0-6883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.
          *Build*: 6.6.0-6883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.

          Pcap files [https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip]
          drigby Dave Rigby made changes -
          Assignee Dave Rigby [ drigby ] Daniel Owen [ owend ]
          owend Daniel Owen made changes -
          Assignee Daniel Owen [ owend ] Paolo Cocchi [ paolo.cocchi ]
          owend Daniel Owen made changes -
          Due Date 24/Jul/20
          owend Daniel Owen made changes -
          Due Date 24/Jul/20 22/Jul/20
          paolo.cocchi Paolo Cocchi made changes -
          Assignee Paolo Cocchi [ paolo.cocchi ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Environment 6.6.0-6883 Enterprise Edition
          couchbase-transactions-1.1.0-SNAPSHOT
          java-client-3.0.6-SNAPSHOT
          6.6.0-7883 Enterprise Edition
          couchbase-transactions-1.1.0-SNAPSHOT
          java-client-3.0.6-SNAPSHOT
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Description *Build*: 6.6.0-6883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.

          Pcap files [https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip]
          *Build*: 6.6.0-7883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.

          Pcap files [https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip]
          ritam.sharma Ritam Sharma made changes -
          Assignee Ashwin Govindarajulu [ ashwin.govindarajulu ] Paolo Cocchi [ paolo.cocchi ]
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Description *Build*: 6.6.0-7883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.

          Pcap files [https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip]
          *Build*: 6.6.0-7883

          *Scenario*:
           # 4 node cluster, 2 Couchbase bucket (replica=2)
          {noformat}+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
          | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
          | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
          | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
          | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
          +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+{noformat}
          {noformat}+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
          | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
          +----------+---------+----------+------------+-----+-------+------------+----------+-----------+
          {noformat}

           # Load few docs into the buckets using transaction
           # Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
           # Start parallel doc_loading using transaction during rebalance phase and validate

          *Observation*:

          At the end of the rebalance during data validation seeing keys missing from the bucket.

          Pcap files [https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip]

          *Test case to run:*
          {noformat}
          guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i nodes.ini -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True -p rerun=False,get-cbcollect-info=False -m rest'{noformat}
          lynn.straus Lynn Straus made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          paolo.cocchi Paolo Cocchi made changes -
          Assignee Paolo Cocchi [ paolo.cocchi ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
          paolo.cocchi Paolo Cocchi made changes -
          Resolution User Error [ 10100 ]
          Status Open [ 1 ] Resolved [ 5 ]
          paolo.cocchi Paolo Cocchi made changes -
          Component/s couchbase-bucket [ 10173 ]
          Component/s test-execution [ 10231 ]
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Attachment KeyMissing Wireshark.png [ 101804 ]
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Attachment key_missing.test [ 101805 ]
          ritam.sharma Ritam Sharma made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            Assignee:
            ashwin.govindarajulu Ashwin Govindarajulu
            Reporter:
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Due:
              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty