Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-1813
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
-
KV 2023-4, March-June 24
Description
Steps To Recreate:
- Create a 4 node cluster
- Create a magma bucket(replicas =2)
- Load 100000 docs(doc_size = 256 bytes)
- While data loading is going on failover a node and rebalance afterwards
- Add back the above node
- Observed ERROR 257: Exception occurred during packet execution. Closing connection [
{"ip":"127.0.0.1","port":48964}
-
{"ip":"127.0.0.1","port":11209}(System, @ns_server) ]: Monotonic<m> (PassiveStream(vb:871 eq_dcpq:replication:ns_1@172.23.216.146->ns_1@172.23.216.142:default)::cur_snapshot_start) invariant failed: new value (1114) breaks invariant on current value (1120).
Below error is from node 172.23.216.142
Error:
2023-11-21T03:59:14.138587+00:00 ERROR 257: Exception occurred during packet execution. Closing connection [ {"ip":"127.0.0.1","port":48964} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]: Monotonic<m> (PassiveStream(vb:871 eq_dcpq:replication:ns_1@172.23.216.146->ns_1@172.23.216.142:default)::cur_snapshot_start) invariant failed: new value (1114) breaks invariant on current value (1120). Cookies: [{"aiostat":"success","ewouldblock":false,"packet":{"bodylen":37,"cas":0,"datatype":"raw","extlen":1,"extras":{"version":0},"keylen":0,"magic":"ClientRequest","opaque":254,"opcode":"DCP_SNAPSHOT_MARKER","value":{"end_seqno":"1128","flags":["raw: 0x00000005","Memory","Checkpoint"],"high_completed_seqno":"0","max_visible_seqno":"1128","start_seqno":"1114"},"vbucket":871},"refcount":1,"started":"1002862086048103 (207 us ago)","throttled":false}] Exception thrown from: ["#0 /opt/couchbase/bin/memcached() [0x400000+0x144364]","#1 /opt/couchbase/bin/memcached() [0x400000+0x5e8bb5]","#2 /opt/couchbase/bin/memcached() [0x400000+0x5ea3d1]","#3 /opt/couchbase/bin/memcached() [0x400000+0x5ee956]","#4 /opt/couchbase/bin/memcached() [0x400000+0x5dc8ac]","#5 /opt/couchbase/bin/memcached() [0x400000+0x5dc9ed]","#6 /opt/couchbase/bin/memcached() [0x400000+0x3419e0]","#7 /opt/couchbase/bin/memcached() [0x400000+0x1dccb3]","#8 /opt/couchbase/bin/memcached() [0x400000+0x2a8a89]","#9 /opt/couchbase/bin/memcached() [0x400000+0x228f0b]","#10 /opt/couchbase/bin/memcached() [0x400000+0x20f8c0]","#11 /opt/couchbase/bin/memcached() [0x400000+0x2147b4]","#12 /opt/couchbase/bin/memcached() [0x400000+0x21e9d7]","#13 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f0ee2ef4000+0xf84e]","#14 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f0ee2ef4000+0x18b99]","#15 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x357) [0x7f0ee2ef4000+0x19287]","#16 /opt/couchbase/bin/memcached() [0x400000+0x97bcc6]","#17 /opt/couchbase/bin/memcached() [0x400000+0x97c1ce]","#18 /opt/couchbase/bin/memcached() [0x400000+0x97e698]","#19 /opt/couchbase/bin/memcached() [0x400000+0x1cf0d9]","#20 /opt/couchbase/bin/memcached() [0x400000+0x8ce91c]","#21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f0ee213f000+0xe4aa3]","#22 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f0ee289c000+0x7ea7]","#23 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f0ee1e00000+0xfba2f]"]
|
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-durability-vset00-00-tunable_rebalance_swap_majority_6.5_P0/testexec.10069.ini process_concurrency=8,sdk_timeout=60,num_items=100000,doc_size=256,GROUP=P0;durability,EXCLUDE_GROUP=non_dgm;not_for_majority,durability=MAJORITY,active_resident_threshold=70,get-cbcollect-info=True,infra_log_level=info,log_level=info,upgrade_version=7.6.0-1813,sirius_url=http://172.23.120.103:4000 -t rebalance_new.swaprebalancetests.SwapRebalanceFailedTests.test_add_back_failed_node,nodes_init=4,replicas=2,standard_buckets=1,num-swap=1,swap-orchestrator=True,GROUP=P0;durability'
|
Attachments
Issue Links
- is duplicated by
-
MB-60475 ERROR 248: Exception occurred during packet execution. Monotonic<m> (PassiveStream(vb:317 eq_dcpq:replication:ns_1@172.23.217.74->ns_1@172.23.217.73:default)::cur_snapshot_start) invariant failed: new value (517) breaks invariant on current value (526)
- Closed
- is triggering
-
MB-60477 Review ActiveStream::nextSnapStart semantic
- Open
For Gerrit Dashboard: MB-59759 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
203824,5 | MB-59759: Do not send snap_start_seqno lower than the requested | master | kv_engine | Status: ABANDONED | +2 | +1 |
203990,2 | MB-59759: Do not send snap_start_seqno lower than the requested | trinity | kv_engine | Status: MERGED | +2 | +1 |
204134,1 | Merge remote-tracking branch 'couchbase/trinity' into trunk | master | kv_engine | Status: MERGED | +2 | +1 |