[System Test] Multiple error are seen while running component system tests

2021-09-06 01:09:08,827 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+--

2021-09-06 01:09:09,482 - systestmon - WARNING - *** 2 occurences of fata keyword found on 172.23.104.21 ***

2021-09-06 01:09:09,483 - systestmon - DEBUG - 2021-09-06T00:52:09.922-07:00 FATA CBAS.util.ExitUtil [Executor-75:ClusterController : WaitForCompletionForJobId: JID:0.1844] JVM halting with status 22 (halting thread Thread[Executor-75:ClusterController : WaitForCompletionForJobId: JID:0.1844,10,main], interrupted false)

2021-09-06 01:09:09,483 - systestmon - DEBUG - 2021-09-06T00:52:10.276-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 

2021-09-06 01:09:12,518 - systestmon - WARNING - There have been more occurences of keyword fata in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 05:58:18,213 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+-- 2021-09-06 05:58:18,896 - systestmon - WARNING - *** 3 occurences of fata keyword found on 172.23.104.17 *** 2021-09-06 05:58:18,896 - systestmon - DEBUG - 2021-09-06T05:26:25.231-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.9604:TAID:TID:ANID:ODID:1:0:20:0:0] Restarting process to ensure data integrity 2021-09-06 05:58:18,897 - systestmon - DEBUG - 2021-09-06T05:26:25.232-07:00 FATA CBAS.util.ExitUtil [SA:JID:9.9604:TAID:TID:ANID:ODID:1:0:20:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:9.9604:TAID:TID:ANID:ODID:1:0:20:0:0,5,main], interrupted true) 2021-09-06 05:58:18,897 - systestmon - DEBUG - 2021-09-06T05:26:26.262-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 

2021-09-06 08:14:11,650 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for HYR0 --+--+--+--+--

2021-09-06 08:14:12,486 - systestmon - WARNING - *** 1 occurences of HYR0 keyword found on 172.23.104.17 ***

2021-09-06 08:14:12,487 - systestmon - DEBUG - org.apache.hyracks.api.exceptions.HyracksDataException: HYR0003: Failure on node 3c01f8f47af9b8b770658c398e3c88c8

2021-09-06 08:14:15,285 - systestmon - WARNING - There have been more occurences of keyword HYR0 in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 09:27:09,057 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+-- 2021-09-06 09:27:09,829 - systestmon - WARNING - *** 7 occurences of fata keyword found on 172.23.104.17 *** 2021-09-06 09:27:09,830 - systestmon - DEBUG - 2021-09-06T08:56:55.431-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.24256:TAID:TID:ANID:ODID:1:0:21:0:0] Restarting process to ensure data integrity 2021-09-06 09:27:09,830 - systestmon - DEBUG - 2021-09-06T08:56:55.432-07:00 FATA CBAS.util.ExitUtil [SA:JID:9.24256:TAID:TID:ANID:ODID:1:0:21:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:9.24256:TAID:TID:ANID:ODID:1:0:21:0:0,5,main], interrupted true) 2021-09-06 09:27:09,830 - systestmon - DEBUG - 2021-09-06T08:56:55.455-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.24256:TAID:TID:ANID:ODID:1:0:18:0:0] Restarting process to ensure data integrity 2021-09-06 09:27:09,830 - systestmon - DEBUG - 2021-09-06T08:56:56.585-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 2021-09-06 09:27:09,830 - systestmon - WARNING - There have been more occurences of keyword fata in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 09:27:11,467 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for HYR0 --+--+--+--+--

2021-09-06 09:27:12,348 - systestmon - WARNING - *** 1 occurences of HYR0 keyword found on 172.23.104.17 ***

2021-09-06 09:27:12,348 - systestmon - DEBUG - org.apache.hyracks.api.exceptions.HyracksDataException: HYR0003: Failure on node 3c01f8f47af9b8b770658c398e3c88c8

2021-09-06 10:39:34,338 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for HYR0 --+--+--+--+-- 2021-09-06 10:39:35,041 - systestmon - WARNING - *** 3 occurences of HYR0 keyword found on 172.23.104.17 *** 2021-09-06 10:39:35,042 - systestmon - DEBUG - org.apache.hyracks.api.exceptions.HyracksDataException: HYR0003: Failure on node 3c01f8f47af9b8b770658c398e3c88c8 2021-09-06 10:39:35,042 - systestmon - DEBUG - org.apache.hyracks.api.exceptions.HyracksDataException: HYR0003: Failure on node 39b9716f0886a25ce32334cfa08ae845 2021-09-06 10:39:35,042 - systestmon - DEBUG - org.apache.hyracks.api.exceptions.HyracksDataException: HYR0003: Failure on node 39b9716f0886a25ce32334cfa08ae845 2021-09-06 10:39:35,589 - systestmon - WARNING - There have been more occurences of keyword HYR0 in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 15:30:37,741 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+-- 2021-09-06 15:30:38,472 - systestmon - WARNING - *** 10 occurences of fata keyword found on 172.23.104.17 *** 2021-09-06 15:30:38,472 - systestmon - DEBUG - 2021-09-06T15:16:02.417-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.73920:TAID:TID:ANID:ODID:1:0:8:0:0] Restarting process to ensure data integrity 2021-09-06 15:30:38,472 - systestmon - DEBUG - 2021-09-06T15:16:02.418-07:00 FATA CBAS.util.ExitUtil [SA:JID:9.73920:TAID:TID:ANID:ODID:1:0:8:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:9.73920:TAID:TID:ANID:ODID:1:0:8:0:0,5,main], interrupted true) 2021-09-06 15:30:38,472 - systestmon - DEBUG - 2021-09-06T15:16:03.165-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 2021-09-06 15:30:39,056 - systestmon - WARNING - There have been more occurences of keyword fata in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 19:11:06,103 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+-- 2021-09-06 19:11:06,961 - systestmon - WARNING - *** 20 occurences of fata keyword found on 172.23.104.17 *** 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:38:43.202-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110154:TAID:TID:ANID:ODID:1:0:15:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:38:43.203-07:00 FATA CBAS.util.ExitUtil [SA:JID:9.110154:TAID:TID:ANID:ODID:1:0:15:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:9.110154:TAID:TID:ANID:ODID:1:0:15:0:0,5,main], interrupted true) 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:38:43.915-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:39:30.307-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:13:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:39:30.308-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:10:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,961 - systestmon - DEBUG - 2021-09-06T18:39:30.308-07:00 FATA CBAS.util.ExitUtil [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:13:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:13:0:0,5,main], interrupted true) 2021-09-06 19:11:06,962 - systestmon - DEBUG - 2021-09-06T18:39:30.309-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:15:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,962 - systestmon - DEBUG - 2021-09-06T18:39:30.311-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:12:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,962 - systestmon - DEBUG - 2021-09-06T18:39:30.311-07:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:9.110164:TAID:TID:ANID:ODID:1:0:8:0:0] Restarting process to ensure data integrity 2021-09-06 19:11:06,962 - systestmon - DEBUG - 2021-09-06T18:39:30.961-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 2021-09-06 19:11:07,514 - systestmon - WARNING - There have been more occurences of keyword fata in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 02:21:54,224 - systestmon - INFO - --+--+--+--+-- Parsing logs for crash component looking for exited with status --+--+--+--+-- 2021-09-06 02:21:58,789 - systestmon - WARNING - *** 1 occurences of exited with status keyword found on 172.23.104.23 *** 2021-09-06 02:21:58,789 - systestmon - DEBUG - [user:info,2021-09-06T01:28:52.170-07:00,ns_1@172.23.104.23:<0.27102.5>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 137. Restarting. Messages: 2021-09-06 02:22:01,143 - systestmon - WARNING - *** 1 occurences of exited with status keyword found on 172.23.97.82 *** 2021-09-06 02:22:01,143 - systestmon - DEBUG - [user:info,2021-09-06T02:20:10.050-07:00,ns_1@172.23.97.82:<0.8566.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 137. Restarting. Messages: 2021-09-06 02:22:01,694 - systestmon - WARNING - There have been more occurences of keyword exited with status in the logs since the last iteration. Hence performing a cbcollect.

2021-09-06 04:46:47,043 - systestmon - WARNING - *** 1 occurences of exited with status keyword found on 172.23.104.21 *** 2021-09-06 04:46:47,043 - systestmon - DEBUG - [user:info,2021-09-06T04:45:15.672-07:00,ns_1@172.23.104.21:<0.21068.29>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 137. Restarting. Messages: 2021-09-06 04:46:48,162 - systestmon - WARNING - *** 1 occurences of exited with status keyword found on 172.23.104.23 *** 2021-09-06 04:46:50,598 - systestmon - WARNING - *** 2 occurences of exited with status keyword found on 172.23.97.77 *** 2021-09-06 04:46:50,599 - systestmon - DEBUG - [user:info,2021-09-06T04:07:32.644-07:00,ns_1@172.23.97.77:<0.5761.22>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 137. Restarting. Messages: 2021-09-06 04:46:51,120 - systestmon - WARNING - *** 1 occurences of exited with status keyword found on 172.23.97.82 *** 2021-09-06 04:46:51,120 - systestmon - WARNING - There have been more occurences of keyword exited with status in the logs since the last iteration. Hence performing a cbcollect.