Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
None
-
7.6.0
-
Operating System : Debian GNU/Linux 12 (bookworm)
Couchbase Enterprise Edition 7.6.0-1907
-
Untriaged
-
Linux x86_64
-
-
0
-
Unknown
Description
Steps to reproduce
- Created a 4 node kv cluster
- Created 10 buckets with different configurations
- Created 5 scopes per bucket and 20 collections per scope
- Loaded data onto each collection (Around 4000 docs onto each collection)
- Added in another kv node and started a rebalance
- Stopped the rebalance
- Started the rebalance again - Rebalance fails at this point, ref :
MB-59828 - Rebalance was re-tried - Rebalance succeeds
- Removed one of the nodes
- Started a rebalance
Rebalance hangs for 50+ hours
Observing a lot of slow operations in the memcached logs
2023-12-15T02:22:07.964205-08:00 WARNING 589: Slow operation: {"bucket":"bucket10","cid":"127.0.0.1:59500/0","command":"SEQNO_PERSISTENCE","duration":"30 s","packet":{"bodylen":8,"cas":0,"datatype":"raw","extlen":8,"extras":{"seqno":"1706"},"keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"SEQNO_PERSISTENCE","vbucket":603},"peer":{"ip":"127.0.0.1","port":59500},"response":"Temporary failure","trace":"execute=14000056536655469:158 execute=14000086600349370:33 request=14000056536655469:30063727","worker_tid":139900734097088} |
2023-12-15T02:22:07.964440-08:00 WARNING 209: Slow operation: {"bucket":"bucket10","cid":"127.0.0.1:36032/0","command":"SEQNO_PERSISTENCE","duration":"30 s","packet":{"bodylen":8,"cas":0,"datatype":"raw","extlen":8,"extras":{"seqno":"1906"},"keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"SEQNO_PERSISTENCE","vbucket":812},"peer":{"ip":"127.0.0.1","port":36032},"response":"Temporary failure","trace":"execute=14000056536776112:50 execute=14000086600670262:9 request=14000056536776112:30063904","worker_tid":139900725704384} |
2023-12-15T02:22:07.965981-08:00 INFO (bucket10) Added SeqnoPersistence request for vb:812, requested-seqno:1906, high-seqno: 0, persisted-seqno:0, cookie:0x7f3d32d87a80, timeout:30000 |
2023-12-15T02:22:07.966050-08:00 INFO (bucket10) Added SeqnoPersistence request for vb:603, requested-seqno:1706, high-seqno: 912, persisted-seqno:912, cookie:0x7f3d32d87900, timeout:30000 |
2023-12-15T02:22:38.011545-08:00 WARNING (bucket10) Notified SeqnoPersistence timeout for vb:812 Check for: 1906, Persisted upto: 0, cookie 0x7f3d32d87a80 |
2023-12-15T02:22:38.011579-08:00 WARNING (bucket10) Notified SeqnoPersistence timeout for vb:603 Check for: 1706, Persisted upto: 912, cookie 0x7f3d32d87900 |
2023-12-15T02:22:38.011812-08:00 WARNING 589: Slow operation: {"bucket":"bucket10","cid":"127.0.0.1:59500/0","command":"SEQNO_PERSISTENCE","duration":"30 s","packet":{"bodylen":8,"cas":0,"datatype":"raw","extlen":8,"extras":{"seqno":"1906"},"keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"SEQNO_PERSISTENCE","vbucket":812},"peer":{"ip":"127.0.0.1","port":59500},"response":"Temporary failure","trace":"execute=14000086602240452:120 execute=14000116647943646:24 request=14000086602240452:30045727","worker_tid":139900734097088} |
2023-12-15T02:22:38.012053-08:00 WARNING 209: Slow operation: {"bucket":"bucket10","cid":"127.0.0.1:36032/0","command":"SEQNO_PERSISTENCE","duration":"30 s","packet":{"bodylen":8,"cas":0,"datatype":"raw","extlen":8,"extras":{"seqno":"1706"},"keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"SEQNO_PERSISTENCE","vbucket":603},"peer":{"ip":"127.0.0.1","port":36032},"response":"Temporary failure","trace":"execute=14000086602325971:47 execute=14000116648262741:12 request=14000086602325971:30045949","worker_tid":139900725704384} |
