KV node rebalance in failed with mover_crashed, unexpected_exit error

./testrunner -i /tmp/testexec.111655.ini -p get-cbcollect-info=True,reset_services=False,java_sdk_client=True,bucket_storage=magma,GROUP=magma_rebalance_20dgm_set2,get-cbcollect-info=True,enable_cdc=True -t eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations,nodes_init=6,services_init=kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,,reset_services=True,bucket_storage=magma,number_of_documents=640000,bucket_size=512,eviction_policy=fullEviction,dgm_run=True,batch_size=100000,java_sdk_client=True,GROUP=magma_rebalance_20dgm_set2

2023-08-08T07:57:15.599-07:00, ns_orchestrator:0:info:message(ns_1@172.23.122.170) - Starting rebalance, KeepNodes = ['ns_1@172.23.107.111','ns_1@172.23.107.112',

                                 'ns_1@172.23.107.132','ns_1@172.23.107.149',

                                 'ns_1@172.23.108.188','ns_1@172.23.122.170',

                                 'ns_1@172.23.122.98'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d8120656d62ddc968739aed19b7aa82d

2023-08-08T08:00:04.198-07:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.122.170) - Worker <0.4160.5> (for action {move,{507,

                                     ['ns_1@172.23.122.170',

                                      'ns_1@172.23.122.98'],

                                     ['ns_1@172.23.108.188',

                                      'ns_1@172.23.122.98'],

                                     []}}) exited with reason {unexpected_exit,

                                                               {'EXIT',

                                                                <0.4190.5>,

                                                                {socket_closed,

                                                                 {gen_server,

                                                                  call,

                                                                  [{'janitor_agent-metadata',

                                                                    'ns_1@172.23.122.170'},

                                                                   {if_rebalance,

                                                                    <0.3890.5>,

                                                                    {wait_dcp_data_move,

                                                                     ['ns_1@172.23.108.188',

                                                                      'ns_1@172.23.122.98'],

                                                                     507}},

                                                                   infinity]}}}}

2023-08-08T08:00:04.218-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.122.170) - Rebalance exited with reason {mover_crashed,

                              {unexpected_exit,

                               {'EXIT',<0.4190.5>,

                                {socket_closed,

                                 {gen_server,call,

                                  [{'janitor_agent-metadata',

                                    'ns_1@172.23.122.170'},

                                   {if_rebalance,<0.3890.5>,

                                    {wait_dcp_data_move,

                                     ['ns_1@172.23.108.188',

                                      'ns_1@172.23.122.98'],

                                     507}},

                                   infinity]}}}}}.

Rebalance Operation Id = d8120656d62ddc968739aed19b7aa82d