[System Test] Rebalance failure - i/o timeout - LSS data copy failed :plasma log copy failed

[user:error,2024-02-29T14:17:16.404-08:00,ns_1@172.23.97.67:<0.22946.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index,

                              {worker_died,

                               {'EXIT',<0.24322.242>,

                                {task_failed,rebalance,

                                 {service_error,

                                  <<"shard copy aborted: LSS data copy failed :plasma log copy failed, dst path :https://172.23.97.108:9104//plasma_storage_v1/02d5352ee78ee8950776ff00570f75b4_ShardToken39_f4_18_cd_c5_10_e_90/16732099842366582290/shards/shard16732099842366582290/data error :log segment copy failed, src:/data/@2i/shards/shard16732099842366582290/data/log.00000000000000.data dst:https://172.23.97.108:9104//plasma_storage_v1/02d5352ee78ee8950776ff00570f75b4_ShardToken39_f4_18_cd_c5_10_e_90/16732099842366582290/shards/shard16732099842366582290/data/log.00000000000000.data off:1047151391, written: 44911841 error: FileTransfer error, dstPath :https://172.23.97.108:9104//plasma_storage_v1/02d5352ee78ee8950776ff00570f75b4_ShardToken39_f4_18_cd_c5_10_e_90/16732099842366582290/shards/shard16732099842366582290/data/log.00000000000000.data error :doRPCReadReq:read tcp4 172.23.97.108:9104->172.23.106.171:36362: i/o timeout, shardId :16732099842366582290">>}}}}}.

Rebalance Operation Id = c4e2904c97134a5d7d28926f650bf88a