Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19576

[FTS] OOMs noted during indexing + querying[retrieving 10K docs]

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • cbft
    • Untriaged
    • Unknown

    Description

      Build
      4.5.0-2478

      Testcase
      ./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_query_in_parallel,items=1000000,num_queries=200,query_max_matches=10000,GROUP=P1

      Testlog - https://gist.github.com/arunapiravi/43cb9e413aafde826adb5cfc19b5de03

      [2016-05-09 00:55:08,030] - [task:1099] INFO - ------------------------------------------------------------------ Query # 90 -----------------------------------------------------------------
      [2016-05-09 00:55:08,045] - [fts_base:1132] INFO - Running query {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000} on node: 172.23.107.234:
      [2016-05-09 00:56:08,365] - [rest_client:775] ERROR - http://172.23.107.234:8094/api/index/default_index/query error 400 reason: status: 400, content: rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found 
       
       rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found 
      [2016-05-09 00:56:08,365] - [fts_base:689] ERROR - Error running query: 'NoneType' object is not iterable
      [2016-05-09 00:56:08,365] - [task:1103] INFO - Status: {}
      [2016-05-09 00:56:08,365] - [task:1127] INFO - FTS hits for query: {"field": "email", "match": "gallia@mcdiabetes.com"} is -1 (took 0.0ms)
      

      On .145's syslog, I see 2 OOMs-

      May  9 00:24:46 localhost kernel: Out of memory: Kill process 14040 (cbft) score 331 or sacrifice child
      :
      :
      May  9 00:55:29 localhost kernel: Out of memory: Kill process 14772 (cbft) score 288 or sacrifice child
      

      Attaching cbcollect info.

      Attachments

        For Gerrit Dashboard: MB-19576
        # Subject Branch Project Status CR V

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty