fatal error: runtime: out of memory error seen in query log though vm still has memory

Error reported in query log:

016-04-26T11:56:55.657-07:00 [Info] GSIC[default/simple_table-1461696995274567730] 00a97f99-ea7c-40a1-a983-4b247a8dc373 new backfill file ... /tmp/scan-backfill27726835790229
2016-04-26T11:56:55.688-07:00 [Info] GSIC[default/simple_table-1461696995274567730] Scan(00a97f99-ea7c-40a1-a983-4b247a8dc373) removing backfill file /tmp/scan-backfill27726420688782 ...
2016-04-26T11:56:55.705-07:00 [Info] GSIC[default/simple_table-1461696995274567730] Scan(00a97f99-ea7c-40a1-a983-4b247a8dc373) removing backfill file /tmp/scan-backfill27726835790229 ...
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x19ef3b7)
/usr/local/go/src/runtime/panic.go:491 +0xad fp=0x7f2c54df9c78 sp=0x7f2c54df9c48
runtime.SysMap(0xc20bf00000, 0x800000000, 0x7f2c698d5000, 0x1a20958)
/usr/local/go/src/runtime/mem_linux.c:149 +0x98 fp=0x7f2c54df9cb0 sp=0x7f2c54df9c78
runtime.MHeap_SysAlloc(0x1a26040, 0x800000000, 0x434f82)
/usr/local/go/src/runtime/malloc.c:284 +0x124 fp=0x7f2c54df9d08 sp=0x7f2c54df9cb0
MHeap_Grow(0x1a26040, 0x400000, 0x0)
/usr/local/go/src/runtime/mheap.c:398 +0x58 fp=0x7f2c54df9d48 sp=0x7f2c54df9d08
MHeap_AllocSpanLocked(0x1a26040, 0x400000, 0x20)
/usr/local/go/src/runtime/mheap.c:298 +0x365 fp=0x7f2c54df9d88 sp=0x7f2c54df9d48
mheap_alloc(0x1a26040, 0x400000, 0x7f0100000000, 0x20)
/usr/local/go/src/runtime/mheap.c:190 +0x121 fp=0x7f2c54df9db0 sp=0x7f2c54df9d88
runtime.MHeap_Alloc(0x1a26040, 0x400000, 0x10100000000, 0xc20a8764e0)
/usr/local/go/src/runtime/mheap.c:240 +0x66 fp=0x7f2c54df9de8 sp=0x7f2c54df9db0
runtime.largeAlloc_m()
/usr/local/go/src/runtime/malloc.c:388 +0xce fp=0x7f2c54df9e38 sp=0x7f2c54df9de8
runtime.onM(0xc2080ec000)
/usr/local/go/src/runtime/asm_amd64.s:257 +0x68 fp=0x7f2c54df9e40 sp=0x7f2c54df9e38
runtime.mstart()
/usr/local/go/src/runtime/proc.c:818 fp=0x7f2c54df9e48 sp=0x7f2c54df9e40

goroutine 124 [running]:
runtime.switchtoM()
/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc20be11fb0 sp=0xc20be11fa8
runtime.mallocgc(0x800000000, 0xf2c7a0, 0xc200000000, 0xc20b5d0000)
/usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xc20be12060 sp=0xc20be11fb0
runtime.newarray(0xf2c7a0, 0x100000000, 0xc20b5d0000)
/usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc20be12098 sp=0xc20be12060
runtime.makeslice(0xe904c0, 0x0, 0x100000000, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc20be120e0 sp=0xc20be12098
github.com/couchbase/query/planner.constrainSpans(0xc20bd00000, 0x10000, 0x10000, 0xc20bd80000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/planner/sarg_and.go:71 +0x25f fp=0xc20be121e8 sp=0xc20be120e0
github.com/couchbase/query/planner.func·010(0x7f2c69910f10, 0xc2082cc0a0, 0xc20bd00000, 0x10000, 0x10000, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/planner/sarg_and.go:44 +0x345 fp=0xc20be122d8 sp=0xc20be121e8
github.com/couchbase/query/planner.(*sargBase).VisitField(0xc20831ea10, 0xc2082cc0a0, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/query/planner/sarg_base.go:198 +0x92 fp=0xc20be12358 sp=0xc20be122d8

Test which fails:
cen006-n1ql-vset03-01-rqg-functions-numeric-string-datetime-simple-table-primary-secondary-tests
./testrunner -i ${resource_tuq} -p gsi_type=${gsi_type} -t rqg.test_rqg.RQGTests.test_rqg_concurrent,test_file_path=b/resources/rqg/simple_table_db/query_tests_using_templates/query_50000_functions_numeric_string_datetime.txt.zip,database=simple_table_db,reset_database=True,concurreny_count=10,index_quota_percent=30,password=password,use_mysql=True,replicas=0,create_secondary_indexes=True,run_query_with_primary=True,record_failure=False,build_index_batch_size=1000,total_queries=${total_queries},skip_cleanup=True

Error reported by test:
This test has always run 10000 queries on an average successfully and 50000 for special cases on the same vm. But recently it started failing with following error:
Reason :: unable to reach the host @ 172.23.106.34

Memory usage:
[root@slv-s20301_c logs]# free -m
total used free shared buffers cached
Mem: 15887 3951 11936 0 109 257
-/+ buffers/cache: 3585 12302
Swap: 10575 880 9695

Disk space usage:
[root@slv-s20301_c logs]# df -k .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-lv_root
35146488 27478332 5876156 83% /

Uploading logs.