Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.0
-
None
-
Triaged
-
No
-
KV-Engine Mad-Hatter GA, KV Sprint 2020-January
Description
Related to MB-37161
Attempting to process a DCP_SEQNO_ACK before the first chain is set removes the connection, and so will result in rebalance failure.
Dependent on what ns_server does we might never manage to get past this, because re-triggering the rebalance could hit exactly the same situation.
2019-12-04T17:57:52.233463-08:00 ERROR 56: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 172.23.121.225:47004 - 172.23.121.226:11209 (<ud>@ns_server</ud>) ]","engine_storage":"0x00007fca9c06a010","ewouldblock":false,"packet":{"bodylen":8,"cas":0,"datatype":"raw","extlen":8,"key":"<ud></ud>","keylen":0,"magic":"ClientRequest","opaque":1,"opcode":"DCP_SEQNO_ACKNOWLEDGED","vbucket":854},"refcount":1}] - closing connection ([ 172.23.121.225:47004 - 172.23.121.226:11209 (<ud>@ns_server</ud>) ]): ActiveDurabilityMonitor::State::processSeqnoAck vb:854 FirstChain not set
|
2019-12-04T17:57:52.233542-08:00 INFO 56: (No Engine) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.226->ns_1@172.23.121.225:default - Removing connection [ 172.23.121.225:47004 - 172.23.121.226:11209 (<ud>@ns_server</ud>) ]
|
2019-12-04T17:57:52.233568-08:00 WARNING 56: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.226->ns_1@172.23.121.225:default - (vb:854) Stream closing, sent until seqno 2 remaining items 0, reason: The stream closed early because the conn was disconnected
|
... all other streams on this connection also close ...
|
...
|
# ns_server sets topology on all vBuckets, including vb:854 which encountered the error above:
|
2019-12-04T17:57:56.845170-08:00 INFO (default) VBucket::setState: transitioning vb:854 with high seqno:2 from:active to:active meta:{"topology":[["ns_1@172.23.121.226","ns_1@172.23.121.225"]]}
|
Attachments
Issue Links
- relates to
-
MB-37161 [Upgrade] failover upgrade failed during rebalance last upgrade node in cluster.
-
- Closed
-
For Gerrit Dashboard: MB-37188 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
119079,6 | MB-37188: Tolerate seqno ack before topology received | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
120379,1 | Merge branch 'mad-hatter' | master | kv_engine | Status: MERGED | +2 | +1 |