Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
None
-
Untriaged
-
Unknown
Description
Test Background:
- Cluster config: 4 nodes(1KV, 2 index, 1 n1ql), ubuntu 14.04
- 1 bucket, 100K items, average doc size 1KB
- Query workload roughly 1/sec, test workload gen used
#! /usr/bin/env bash
|
|
URL=http://localhost:9496/query
|
HEADER='-H Content-Type:text/plain'
|
|
run_query(){
|
curl -s -X POST $HEADER --data "$1" $URL &>/dev/null &
|
sleep 1
|
}
|
|
while :
|
do
|
run_query "select * from default limit 1;"
|
done
|
- Primary index defined using views:
cbq> select * from system:indexes;
|
{
|
"requestID": "3659edbc-bd0d-4c8a-b086-fe9cf7ec27ba",
|
"signature": {
|
"*": "*"
|
},
|
"results": [
|
{
|
"indexes": {
|
"datastore_id": "http://127.0.0.1:9003",
|
"id": "#primary",
|
"index_key": [
|
"(meta(`default`).`id`)"
|
],
|
"keyspace_id": "default",
|
"name": "#primary",
|
"namespace_id": "default",
|
"state": "online",
|
"using": "view"
|
}
|
}
|
],
|
"status": "success",
|
"metrics": {
|
"elapsedTime": "369.317088ms",
|
"executionTime": "368.767589ms",
|
"resultCount": 1,
|
"resultSize": 424
|
}
|
}
|
- cbq-engine at startup was using under 50MB and when I let the workload run for 5-10 mins it's me usage grew upto 4.5GB before it panicked and crashed.
mem-usage at startup:
mem-usage before coq-engine crashed:
- I enabled cpu and mem profile flags for cbq-engine. This what I observed
mem-profile:
(pprof) top 10
|
Total: 2479.4 MB
|
416.0 16.8% 16.8% 489.0 19.7% encoding/json.(*decodeState).arrayInterface
|
392.0 15.8% 32.6% 392.0 15.8% bytes.makeSlice
|
291.5 11.8% 44.3% 291.5 11.8% github.com/couchbase/query/execution.(*FinalProject).Copy
|
284.5 11.5% 55.8% 1139.5 46.0% github.com/couchbase/query/execution.(*Sequence).Copy
|
282.5 11.4% 67.2% 282.5 11.4% github.com/couchbase/query/execution.(*Fetch).Copy
|
281.0 11.3% 78.5% 281.0 11.3% github.com/couchbase/query/execution.(*InitialProject).Copy
|
119.9 4.8% 83.4% 119.9 4.8% reflect.unsafe_NewArray
|
111.0 4.5% 87.9% 111.0 4.5% runtime.convT2E
|
47.5 1.9% 89.8% 47.5 1.9% github.com/couchbase/query/datastore.NewSizedIndexConnection
|
46.5 1.9% 91.7% 46.5 1.9% github.com/couchbase/query/execution.NewSequence
|
For some reason I wasn't able to generate svg graph of it(Attaching the pprof files for reference).