Analytics TPCDS Volume test service crash

Running the tpcds volume test with 1.3 TB of data... the following query took around 4 hours then analytics service crashed:

WITH customer_total_return AS (SELECT sr.sr_customer_sk AS ctr_customer_sk, sr.sr_store_sk AS ctr_store_sk, SUM(sr.sr_return_amt) AS ctr_total_return FROM store_returns sr, date_dim dd WHERE sr.sr_returned_date_sk = dd.d_date_sk AND dd.d_year = 2000 GROUP BY sr.sr_customer_sk, sr.sr_store_sk) SELECT c.c_customer_id FROM customer_total_return ctr1, store s, customer c WHERE ctr1.ctr_total_return > (SELECT VALUE AVG(ctr2.ctr_total_return) * 1.2 FROM customer_total_return ctr2 WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)[0] AND s.s_store_sk = ctr1.ctr_store_sk AND s.s_state = "TN" AND ctr1.ctr_customer_sk = c.c_customer_sk ORDER BY c.c_customer_id LIMIT 100;

Here are the logs:

https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.205.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.5.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.57.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.7.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.8.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2019-11-22T180830-ns_1%40172.23.96.9.zip

The job: http://perf.jenkins.couchbase.com/job/oceanus/2511/

Node 172.23.96.5 complained about compaction being unable to proceed due to 0 bytes available on disk. However the disk on that node has at least 80GB of available space: 

[root@oceanus-kv1 ~]# df -h

Filesystem                Size  Used Avail Use% Mounted on

/dev/sda2                 477G  5.3G  471G   2% /

devtmpfs                   16G     0   16G   0% /dev

tmpfs                      16G     0   16G   0% /dev/shm

tmpfs                      16G  931M   15G   6% /run

tmpfs                      16G     0   16G   0% /sys/fs/cgroup

/dev/sda1                 504M  145M  334M  31% /boot

/dev/sdb1                 440G  317G  101G  76% /backups

/dev/mapper/vg_data-data  880G  747G   89G  90% /data

tmpfs                     3.2G     0  3.2G   0% /run/user/0