Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
Cheshire-Cat
-
7.0.0-5270
-
Untriaged
-
1
-
Unknown
Description
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.16593.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=True,log_level=debug,infra_log_level=debug,upgrade_version=7.0.0-5270 -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_on_increasing_replica,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=0,GROUP=P0'
|
Steps:
1.Create a 4 node cluster
2. Create magma bucket with 1 replica
3. Load 5M items
4. Create dummy files to fill up disk
5. Load more data until we see ep_data_write_commit failures
6. Increase replica to 2 and rebalance. Rebalance(Rebalance Operation Id = 5c07cb8daaf60622ed4ae61eca136714) failed due to dcp_wait_for_data_move_failed which is expected.
7. Reduce replica back to 1 and rebalance(Rebalance Operation Id = 6a34f333f226394c1c95d9423f6c83c4)
8. Rebalance failed again:
{u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'5179f995492589ddb3f087f7b2773354', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a009c619d1f5c385ef0d16d7951ff321', u'status': u'notRunning'} - rebalance failed
|
2021-07-03 06:32:58,628 | test | ERROR | pool-5-thread-27 | [rest_client:_rebalance_status_and_progress:1547] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'75461aaf6476facf1bb01c154a88115c', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=058030187ca91e8bc9195bbcd82a3b5b', u'status': u'notRunning'} - rebalance failed
|
2021-07-03 06:32:58,651 | test | ERROR | pool-5-thread-27 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.106.53', u'tstamp': 1625319173934L, u'shortText': u'message', u'serverTime': u'2021-07-03T06:32:53.934Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.4149.2>,\n {{dcp_wait_for_data_move_failed,"default",\n 642,\'ns_1@172.23.106.63\',\n [\'ns_1@172.23.106.56\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,459}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 6a34f333f226394c1c95d9423f6c83c4'}
|
2021-07-03 06:32:58,651 | test | ERROR | pool-5-thread-27 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.106.53', u'tstamp': 1625319173917L, u'shortText': u'message', u'serverTime': u'2021-07-03T06:32:53.917Z', u'text': u'Worker <0.4137.2> (for action {move,{642,\n [\'ns_1@172.23.106.63\'],\n [\'ns_1@172.23.106.63\',\n \'ns_1@172.23.106.56\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.4149.2>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 642,\n \'ns_1@172.23.106.63\',\n [\'ns_1@172.23.106.56\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 459}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'}
|