Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.0
-
Untriaged
-
Unknown
Description
Build
4.5.0-2478
Testcase
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_query_in_parallel,items=1000000,num_queries=200,query_max_matches=10000,GROUP=P1
Testlog - https://gist.github.com/arunapiravi/43cb9e413aafde826adb5cfc19b5de03
[2016-05-09 00:55:08,030] - [task:1099] INFO - ------------------------------------------------------------------ Query # 90 -----------------------------------------------------------------
|
[2016-05-09 00:55:08,045] - [fts_base:1132] INFO - Running query {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000} on node: 172.23.107.234:
|
[2016-05-09 00:56:08,365] - [rest_client:775] ERROR - http://172.23.107.234:8094/api/index/default_index/query error 400 reason: status: 400, content: rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
|
rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
[2016-05-09 00:56:08,365] - [fts_base:689] ERROR - Error running query: 'NoneType' object is not iterable
|
[2016-05-09 00:56:08,365] - [task:1103] INFO - Status: {}
|
[2016-05-09 00:56:08,365] - [task:1127] INFO - FTS hits for query: {"field": "email", "match": "gallia@mcdiabetes.com"} is -1 (took 0.0ms)
|
On .145's syslog, I see 2 OOMs-
May 9 00:24:46 localhost kernel: Out of memory: Kill process 14040 (cbft) score 331 or sacrifice child
|
:
|
:
|
May 9 00:55:29 localhost kernel: Out of memory: Kill process 14772 (cbft) score 288 or sacrifice child
|
Attaching cbcollect info.
Attachments
For Gerrit Dashboard: MB-19576 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
64013,1 | MB-19576 - API CHANGE - commit, not transactions, for ExecuteBatch by default | master | goforestdb | Status: ABANDONED | -2 | 0 |
64038,2 | MB-19576 - when using moss, use forestdb skip_batch config | master | cbft | Status: MERGED | +2 | +1 |
64040,2 | MB-19576 - bump blevex for forestdb skip_batch option | master | manifest | Status: MERGED | +2 | +1 |
64452,1 | MB-19576 - segmentStack doesn't need to know about collection | master | moss | Status: ABANDONED | 0 | 0 |
64453,1 | MB-19576 - refactored out a mergeInto() helper method | master | moss | Status: ABANDONED | 0 | 0 |
64454,1 | MB-19576 - prealloc slice | master | moss | Status: ABANDONED | 0 | 0 |
64455,1 | MB-19576 - renamed param to minSegmentLevel | master | moss | Status: ABANDONED | 0 | 0 |
64456,1 | MB-19576 - exposed some segment methods | master | moss | Status: ABANDONED | 0 | 0 |
64457,1 | MB-19576 - added Segment interface | master | moss | Status: ABANDONED | 0 | 0 |
64458,1 | MB-19576 - simple moss persistence via mmap | master | moss | Status: ABANDONED | 0 | 0 |
64463,1 | MB-19576 more store ref-counting cleanup | master | moss | Status: ABANDONED | 0 | 0 |
64464,1 | MB-19576 renamed to startOrReuseFile() | master | moss | Status: ABANDONED | 0 | 0 |
64465,1 | MB-19576 renamed to loadFooterSegments | master | moss | Status: ABANDONED | 0 | 0 |
64466,1 | MB-19576 - exposed NewSnapshotWrapper() | master | moss | Status: ABANDONED | 0 | 0 |
64467,1 | MB-19576 - handle store.Persist(nil) case | master | moss | Status: ABANDONED | 0 | 0 |
64468,1 | MB-19576 - moss store concurrent dual writers per segment | master | moss | Status: ABANDONED | 0 | 0 |
64469,1 | MB-19576 - added CollectionOptions to StoreOptions | master | moss | Status: ABANDONED | 0 | 0 |
64470,1 | MB-19576 - first attempt at compaction | master | moss | Status: ABANDONED | 0 | 0 |
64471,1 | MB-19576 - before/after file close callbacks | master | moss | Status: ABANDONED | 0 | 0 |
64472,1 | MB-19576 - TestStoreOpsCompactionForce | master | moss | Status: ABANDONED | 0 | 0 |
64473,1 | MB-19576 - fix compaction; TestStoreCompaction added | master | moss | Status: ABANDONED | 0 | 0 |
64474,1 | MB-19576 renamed some compaction concerns | master | moss | Status: ABANDONED | 0 | 0 |
64475,1 | MB-19576 - reread sync'ed footer after compact | master | moss | Status: ABANDONED | 0 | 0 |
64476,1 | MB-19576 - check for 0 kvs bytes / buf bytes | master | moss | Status: ABANDONED | 0 | 0 |
64477,1 | MB-19576 - increase compaction buffer size | master | moss | Status: ABANDONED | 0 | 0 |
64478,1 | MB-19576 - mergeInto() optimizeTail option | master | moss | Status: ABANDONED | 0 | 0 |
64479,1 | MB-19576 - buffer section writer max checked when > 0 | master | moss | Status: ABANDONED | 0 | 0 |
64481,1 | MB-19576 - don't track os page size for now | master | moss | Status: ABANDONED | 0 | 0 |
64482,1 | MB-19576 - header should be shorter than page size | master | moss | Status: ABANDONED | 0 | 0 |
64483,1 | MB-19576 - split out store_footer.go | master | moss | Status: ABANDONED | 0 | 0 |
64484,1 | MB-19576 - split out merger and stats files | master | moss | Status: ABANDONED | 0 | 0 |
64485,1 | MB-19576 - split out segment_stack_merge.go | master | moss | Status: ABANDONED | 0 | 0 |
64487,1 | MB-19576 - bufferedSectionWriter performs async writes | master | moss | Status: ABANDONED | 0 | 0 |
64488,1 | MB-19576 - CompactionBufferPages store option | master | moss | Status: ABANDONED | 0 | 0 |
64489,1 | MB-19576 - check stop err at compaction end | master | moss | Status: ABANDONED | 0 | 0 |
64490,1 | MB-19576 - CompactionSync option | master | moss | Status: ABANDONED | 0 | 0 |
64492,2 | MB-19576 - simple moss store via mmap | master | moss | Status: MERGED | +2 | +1 |
64494,2 | MB-19576 - moss store refactoring and cleanup | master | moss | Status: MERGED | +2 | +1 |
64495,2 | MB-19576 - moss store compaction | master | moss | Status: MERGED | +2 | +1 |