Details
-
Bug
-
Resolution: Fixed
-
Major
-
5.0.0
-
Untriaged
-
-
No
Description
Memcached exists abruptly and causes rebalance failures and causes the ephemeral buckets to go on warmup state when loading data on to the ephemeral buckets.
The ns_server tests are failing/getting struck due to this issue and blocking the test runs.
In the test, we create ephemeral buckets and then load data with multiple clients connecting to the server. The log suggests that memcached exit due to high number of clients being connected. But memcached fails to come back and causes buckets to go on warmup state causing the whole test to fail. Attaching the logs of the cluster for investigation.
http://qa.sc.couchbase.com/job/cen006-nserv-vset05-01-rebalanceinout_P1_set2-ephemeral/2/consoleFull is one of the test suite that is failing due to this issue. Note that this issue is very sporadic and running the same test might not result in the failure. But the issue is seen mostly when we load multiple ephemeral buckets with high number of connections.
Repro steps:
- Clone/update testrunner and create an ini file with the nodes ip.
- Run following test with the ini file create.
- ./testrunner -i <ini file> get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=False,force_kill_memached=False,verify_unacked_bytes=True,std_vbucker_dist=8,chk_max_items=1000,chk_period=30,bucket_type=ephemeral -t rebalance.rebalanceinout.RebalanceInOutTests.test_incremental_rebalance_out_in_with_mutation,init_num_nodes=3,items=400000,standard_buckets=2,sasl_buckets=2,standard_bucket_priority=low:high,sasl_bucket_priority=low:high,zone=2
Attachments
Issue Links
- is duplicated by
-
MB-24214 [Ephemeral] Node in warmup after node is removed the cluster
- Closed