Graceful failover exited with reason {pre_rebalance_janitor_run_failed, "GleamBookUsers0", {error,wait_for_memcached_failed,

Steps:

1. Create a 3 node cluster
2. Create required buckets and collections.
3. Create 100000000 items sequentially
4. Rebalance in with Loading of docs
5. Rebalance reached >20.9991150065% in 130.427999973 seconds. Abort and resume
6. Rebalance reached >50.293161723% in 10.0199999809 seconds. Abort and resume
7. Rebalance reached >71.3359366319% in 10.0399999619 seconds. Abort and resume
8. Rebalance reached >80.420904852% in 10.0360000134 seconds. Abort and resume
9. Sleep 61 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
10. Sleep 72 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
11. Sleep 111 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
12. Sleep 116 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
13. Sleep 93 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
14. Sleep 81 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
15. Sleep 99 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
16. Sleep 111 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
17. Sleep 69 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
18. Sleep 99 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
19. Sleep 99 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
20. Rebalance Out with Loading of docs
21. Rebalance reached >20.0046178537% in 80.1549999714 seconds. Abort and resume
22. Rebalance reached >40.6169263161% in 40.1119999886 seconds. Abort and resume
23. Rebalance reached >62.6878134975% in 10.0249998569 seconds. Abort and resume
24. Rebalance reached >82.2232560402% in 170.571000099 seconds. Abort and resume
25. Sleep 81 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
26. Sleep 63 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
27. Sleep 109 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
28. Sleep 87 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
29. Sleep 120 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
30. Sleep 81 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
31. Sleep 88 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
32. Sleep 116 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
33. Sleep 67 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
34. Sleep 77 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
35. Sleep 96 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
36. Rebalance In_Out with Loading of docs
37. Rebalance reached >21.2970432982% in 330.736999989 seconds. Abort and resume
38. Rebalance reached >40.3188831559% in 140.421999931 seconds. Abort and resume
39. Rebalance reached >60.4980254288% in 140.364000082 seconds. Abort and resume
40. Rebalance reached >80.791803645% in 80.2209999561 seconds. Abort and resume
41. Sleep 62 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
42. Sleep 116 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
43. Sleep 66 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
44. Sleep 91 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
45. Sleep 98 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
46. Sleep 61 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
47. Sleep 81 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
48. Sleep 63 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
49. Sleep 79 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
50. Sleep 88 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
51. Sleep 105 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
52. Swap with Loading of docs
53. Rebalance reached >20.1605890903% in 240.761999846 seconds. Abort and resume
54. Rebalance reached >40.0448430493% in 180.631999969 seconds. Abort and resume
55. Rebalance reached >71.4037063953% in 521.764000177 seconds. Abort and resume
56. Rebalance reached >80.0446428571% in 140.259000063 seconds. Abort and resume
57. Sleep 65 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
58. Sleep 76 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
59. Sleep 70 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
60. Sleep 70 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
61. Sleep 61 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
62. Sleep 99 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
63. Sleep 120 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
64. Sleep 90 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
65. Sleep 67 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
66. Sleep 87 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
67. Sleep 79 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
68. Failover a node and RebalanceOut that node with loading in parallel
69. Sleep 77 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
70. Sleep 86 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
71. Sleep 67 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
72. Sleep 96 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
73. Sleep 99 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
74. Sleep 98 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
75. Sleep 92 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
76. Sleep 95 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
77. Sleep 93 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
78. Sleep 113 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
79. Sleep 114 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
80. Failover a node and FullRecovery that node
81. Sleep 96 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
82. Sleep 63 seconds. Reason: Iteration:1 waiting to kill memc on all nodes
83. Sleep 82 seconds. Reason: Iteration:2 waiting to kill memc on all nodes
84. Sleep 97 seconds. Reason: Iteration:3 waiting to kill memc on all nodes
85. Sleep 80 seconds. Reason: Iteration:4 waiting to kill memc on all nodes
86. Sleep 87 seconds. Reason: Iteration:5 waiting to kill memc on all nodes
87. Sleep 120 seconds. Reason: Iteration:6 waiting to kill memc on all nodes
88. Sleep 115 seconds. Reason: Iteration:7 waiting to kill memc on all nodes
89. Sleep 95 seconds. Reason: Iteration:8 waiting to kill memc on all nodes
90. Sleep 104 seconds. Reason: Iteration:9 waiting to kill memc on all nodes
91. Sleep 94 seconds. Reason: Iteration:10 waiting to kill memc on all nodes
92. Failover a node and DeltaRecovery that node with loading in parallel

Graceful failover exited with reason {pre_rebalance_janitor_run_failed,

"GleamBookUsers0",

{error,wait_for_memcached_failed,

['ns_1@172.23.107.98']}}.

Test Details:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_temp_jython.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=3,replicas=1,skip_cleanup=True,num_items=100000000,num_buckets=2,bucket_names=GleamBook,doc_size=64,bucket_type=membase,compression_mode=off,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=60,num_scopes=10,num_collections=10,doc_ops=expiry,durability=Majority,pc=1,crashes=10,sdk_client_pool=True,get-cbcollect-info=True -m rest'