Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
4.5.0
-
Untriaged
-
Unknown
Description
Build
4.5.0-2478
This testcase earlier failed with OOM errors. Now this problem looks similar to MB-19468. I can consistently reproduce this on recent builds. This bug again is about a narrow window where fts returns 403 error. We do seem to recover and service query requests post this error, so filing as Major.
Testcase
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_query_in_parallel,items=1000000,num_queries=200,query_max_matches=10000,GROUP=P1
Testlog - https://gist.github.com/arunapiravi/43cb9e413aafde826adb5cfc19b5de03
[2016-05-09 00:55:08,030] - [task:1099] INFO - ------------------------------------------------------------------ Query # 90 -----------------------------------------------------------------
|
[2016-05-09 00:55:08,045] - [fts_base:1132] INFO - Running query {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000} on node: 172.23.107.234:
|
[2016-05-09 00:56:08,365] - [rest_client:775] ERROR - http://172.23.107.234:8094/api/index/default_index/query error 400 reason: status: 400, content: rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
|
|
rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
[2016-05-09 00:56:08,365] - [fts_base:689] ERROR - Error running query: 'NoneType' object is not iterable
|
[2016-05-09 00:56:08,365] - [task:1103] INFO - Status: {}
|
[2016-05-09 00:56:08,365] - [task:1127] INFO - FTS hits for query: {"field": "email", "match": "gallia@mcdiabetes.com"} is -1 (took 0.0ms)
|
On .234's fts.log, I see the same error with no signs of cbft crashing before
2016-05-09T00:55:07.924-07:00 [INFO] slow-query: index: default_index, query: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match": "Antonia Cook"}, "size": 10000}, duration: 6.658457009s, err: <nil>
|
2016-05-09T00:55:11.775-07:00 [INFO] cbdatasource: server: 172.23.107.145:11210, uprOpenName: fts:default_index_5442ab3f4ac94e7a-5c9d965f, worker, looping beg, vbucketState: "running" (has 192 vbuckets), 64-95, 128-159, 192-255, 384-415, 480-511
|
2016-05-09T00:55:11.831-07:00 [INFO] cbdatasource: server: 172.23.107.187:11210, uprOpenName: fts:default_index_5442ab3f4ac94e7a-5c9d965f, worker, looping beg, vbucketState: "running" (has 320 vbuckets), 544-735, 768-799, 896-927, 960-1023
|
2016-05-09T00:55:14.531-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
2016-05-09T00:55:35.453-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
2016-05-09T00:55:41.808-07:00 [INFO] cbdatasource: server: 172.23.107.145:11210, uprOpenName: fts:default_index_5442ab3f4ac94e7a-5c9d965f, worker, looping beg, vbucketState: "running" (has 192 vbuckets), 64-95, 128-159, 192-255, 384-415, 480-511
|
2016-05-09T00:55:41.873-07:00 [INFO] cbdatasource: server: 172.23.107.187:11210, uprOpenName: fts:default_index_5442ab3f4ac94e7a-5c9d965f, worker, looping beg, vbucketState: "running" (has 320 vbuckets), 544-735, 768-799, 896-927, 960-1023
|
2016-05-09T00:56:08.332-07:00 [INFO] slow-query: index: default_index, query: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, duration: 1m0.274520917s, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
2016-05-09T00:56:08.359-07:00 [INFO] rest: error code: 400, msg: rest_index: Query, indexName: default_index, requestBody: {"from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "size": 10000}, req: &http.Request{Method:"POST", URL:(*url.URL)(0xc8209c4280), Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json"}, "Authorization":[]string{"Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA=="}, "Accept":[]string{"*/*"}, "User-Agent":[]string{"Python-httplib2/$Rev: 259 $"}, "Accept-Encoding":[]string{"identity"}, "Content-Length":[]string{"223"}}, Body:(*http.body)(0xc82c346340), ContentLength:223, TransferEncoding:[]string(nil), Close:false, Host:"172.23.107.234:8094", Form:url.Values{}, PostForm:url.Values{}, MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.23.106.205:42895", RequestURI:"/api/index/default_index/query", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil)}, err: bleve: QueryBlevePIndexImpl remote client returned status: 403 body: rest_auth: preparePerm, err: pindex not found
|
2016-05-09T00:56:11.830-07:00 [INFO] cbdatasource: server: 172.23.107.145:11210, uprOpenName: fts:default_index_5442ab3f4ac94e7a-5c9d965f, worker, looping beg, vbucketState: "running" (has 192 vbuckets), 64-95, 128-159, 192-255, 384-415, 480-511
|
Attaching cbcollect info.
