Failures loading CH2 data

Trying to load CH2 data for

• 1000 warehouses into
• 4 c4.4xlarge nodes with an 150GB EBS drive using
• 1 c4.8xlarge node as a driver

causes multiple failovers.

Each node in the 4 node cluster has 16 cores, 30GB memory, and 150GB disk. The memory allocation is 25600 MB for data and 3072 MB for index. The client is a separate machine which runs the pytpcc loader. The loader uses upsert statements to load multiple documents at a time.

Using the pytpcc driver to load 1000 warehouses. Was able to load 100 warehouses using 40 client threads successfully. When trying to load more than 100 warehouses, the RAM utilization on the nodes slowly creeps up towards 100% and eventually the node crashes.