You can test it as follows:
- Create an S3 link.
- Create an external dataset on the S3 link, preferably with data for a query to take few seconds (don't finish instantly).
- Query the data (any query will do), upon completion of the query, check the TCP connections open to S3 still, the following is observed:
- Without the fix, you should see the connections (num of connections = num of partitions) still open (for around 1 minute)
- With the fix, you should see all the connections closing within 1-2 seconds of the returning of the query results.
I would recommend increasing the number of partitions and number of files to clearly see large number of connections open, and see all of them closing instantly as the result is returned.
Hussain Towaileb - Can you please help with steps to validate.
CC - Till Westmann