Details
-
Bug
-
Resolution: User Error
-
Critical
-
6.6.0
-
6.6.0-7883 Enterprise Edition
couchbase-transactions-1.1.0-SNAPSHOT
java-client-3.0.6-SNAPSHOT
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Build: 6.6.0-7883
Scenario:
- 4 node cluster, 2 Couchbase bucket (replica=2)
+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
| Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
| 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |
| 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |
| 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |
| 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |
+----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+
+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
| bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |
| bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |
+----------+---------+----------+------------+-----+-------+------------+----------+-----------+
- Load few docs into the buckets using transaction
- Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
- Start parallel doc_loading using transaction during rebalance phase and validate
Observation:
At the end of the rebalance during data validation seeing keys missing from the bucket.
Pcap files https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip
Test case to run:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i nodes.ini -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True -p rerun=False,get-cbcollect-info=False -m rest'
|