Logs will be stored at /data/workspace/cen006-p0-fts-vset00-00-custom-map-rqg/logs/testrunner-16-Aug-15_11-29-36/test_60 ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D+F,GROUP=ALL -t fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping,items=1000,custom_map=True,num_custom_analyzers=1,cm_id=22,num_queries=100,compare_es=True,GROUP=P0 Test Input params: {'get-coredumps': 'True', 'num_custom_analyzers': '1', 'logs_folder': '/data/workspace/cen006-p0-fts-vset00-00-custom-map-rqg/logs/testrunner-16-Aug-15_11-29-36/test_60', 'GROUP': 'ALL', 'compare_es': 'True', 'cluster': 'D+F', 'items': '1000', 'num_queries': '100', 'conf_file': 'conf/fts/py-fts-custmap-rqg-queries.conf', 'num_nodes': 1, 'cluster_name': 'INI_FILE', 'cm_id': '22', 'stop-on-failure': 'False', 'ini': 'INI_FILE.ini', 'case_number': 60, 'get-logs': 'False', 'get-cbcollect-info': 'True', 'custom_map': 'True', 'spec': 'py-fts-custmap-rqg-queries'} [2016-08-15 12:39:35,891] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:39:36,182] - [remote_util:206] INFO - Connected to 172.23.106.72 [2016-08-15 12:39:36,882] - [remote_util:481] INFO - 172.23.106.72 **** The version file /opt/couchbase/ VERSION.txt exists index_query_custom_mapping (fts.stable_topology_fts.StableTopFTS) ... looking for Erlang/Memcached crashes on 172.23.106.72 ... dump files not found on 172.23.106.72 [2016-08-15 12:39:37,104] - [fts_base:1995] INFO - ==== FTSbasetests setup is started for test #60 index_query_custom_mapping ==== [2016-08-15 12:39:37,124] - [fts_base:978] INFO - removing nodes from cluster ... [2016-08-15 12:39:37,141] - [fts_base:980] INFO - cleanup [ip:172.23.106.72 port:8091 ssh_username:root] [2016-08-15 12:39:37,157] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.106.72 [2016-08-15 12:39:37,205] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.106.72:8091 [2016-08-15 12:39:37,217] - [cluster_helper:80] INFO - ns_server @ 172.23.106.72:8091 is running [2016-08-15 12:39:37,221] - [es_base:168] ERROR - http://172.23.107.55:9200/es_index error 404 reason: IndexMissingException[[es_index] missing] {"error":"IndexMissingException[[es_index] missing]","status":404} [2016-08-15 12:39:37,267] - [fts_base:2792] INFO - Created empty index es_index on Elastic Search node with custom standard analyzer(default) [2016-08-15 12:39:37,268] - [fts_base:923] INFO - Initializing Cluster ... [2016-08-15 12:39:38,125] - [rest_client:879] INFO - pools/default params : memoryQuota=2069 [2016-08-15 12:39:38,134] - [rest_client:907] INFO - settings/indexes params : storageMode=forestdb [2016-08-15 12:39:38,142] - [rest_client:795] INFO - settings/web params on 172.23.106.72:8091:username=Administrator&password=password&port=8091 [2016-08-15 12:39:39,207] - [rest_client:1908] INFO - http://172.23.106.72:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-08-15 12:39:39,218] - [rest_client:1930] INFO - 0.01 seconds to create bucket default [2016-08-15 12:39:39,219] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 172.23.106.72 to accept set ops [2016-08-15 12:39:40,480] - [data_helper:295] INFO - creating direct client 172.23.106.72:11210 default [2016-08-15 12:39:40,858] - [data_helper:295] INFO - creating direct client 172.23.106.72:11210 default [2016-08-15 12:39:44,177] - [task:289] INFO - bucket 'default' was created with per node RAM quota: 1569 [2016-08-15 12:39:44,231] - [rest_client:1299] INFO - /diag/eval status on 172.23.106.72:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-08-15 12:39:44,233] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:39:44,527] - [remote_util:206] INFO - Connected to 172.23.106.72 [2016-08-15 12:39:45,212] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-08-15 12:39:46,118] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:39:46,119] - [fts_base:2474] INFO - 0 [2016-08-15 12:39:46,120] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-08-15 12:39:46,933] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:39:46,934] - [fts_base:2474] INFO - 0 [2016-08-15 12:39:46,935] - [fts_base:2205] INFO - {'172.23.106.72': {'crash in forestdb': 0, 'panic:': 0}} [2016-08-15 12:39:46,936] - [fts_base:2004] INFO - ==== FTSbasetests setup is finished for test #60 index_query_custom_mapping ==== {'text': ['name', 'dept'], 'number': ['salary', 'mutated'], 'datetime': ['join_date']} [2016-08-15 12:39:47,003] - [fts_base:611] INFO - { "mapping": { "default_analyzer": "customAnalyzer1", "default_datetime_parser": "dateTimeOptional", "type_field": "type", "default_mapping": { "default_analyzer": "", "dynamic": true, "enabled": false }, "default_type": "_default", "analysis": { "analyzers": { "customAnalyzer1": { "type": "custom", "char_filters": [ "html" ], "tokenizer": "whitespace", "token_filters": [ "shingle" ] } }, "token_filters": { "dict_compound_en": { "dict_token_map": "stop_en", "type": "dict_compound" }, "length": { "max": 5, "type": "length", "min": 3 }, "truncate": { "length": 10, "type": "truncate_token" }, "dict_compound_fr": { "dict_token_map": "articles_fr", "type": "dict_compound" }, "ngram": { "max": 5, "type": "ngram", "min": 3 }, "front_edge_ngram": { "max": 5, "type": "edge_ngram", "back": false, "min": 3 }, "back_edge_ngram": { "max": 5, "type": "edge_ngram", "back": true, "min": 3 }, "shingle": { "output_original": "false", "min": 2, "filler": "", "max": 5, "separator": "", "type": "shingle" }, "keyword_marker_en": { "type": "keyword_marker", "keywords_token_map": "stop_en" } } }, "default_field": "_all", "types": { "emp": { "enabled": true, "dynamic": false, "properties": { "salary": { "enabled": true, "dynamic": false, "properties": {}, "fields": [ { "index": true, "name": "salary", "include_in_all": true, "type": "number", "analyzer": "", "include_term_vectors": true, "store": false } ] }, "dept": { "enabled": true, "dynamic": false, "properties": {}, "fields": [ { "index": true, "name": "dept", "include_in_all": true, "type": "text", "analyzer": "customAnalyzer1", "include_term_vectors": true, "store": false } ] }, "join_date": { "enabled": true, "dynamic": false, "properties": {}, "fields": [ { "index": true, "name": "join_date", "include_in_all": true, "type": "datetime", "analyzer": "", "include_term_vectors": true, "store": false } ] }, "name": { "enabled": true, "dynamic": false, "properties": {}, "fields": [ { "index": true, "name": "name", "include_in_all": true, "type": "text", "analyzer": "customAnalyzer1", "include_term_vectors": true, "store": false } ] }, "mutated": { "enabled": true, "dynamic": false, "properties": {}, "fields": [ { "index": true, "name": "mutated", "include_in_all": true, "type": "number", "analyzer": "", "include_term_vectors": true, "store": false } ] } }, "fields": [] } } } } [2016-08-15 12:39:47,004] - [fts_base:639] INFO - Checking if index already exists ... [2016-08-15 12:39:47,035] - [rest_client:778] ERROR - http://172.23.106.72:8094/api/index/custom_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-08-15 12:39:47,039] - [rest_client:778] ERROR - http://172.23.106.72:8094/api/index/custom_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-08-15 12:39:47,040] - [fts_base:647] INFO - Creating fulltext-index custom_index on 172.23.106.72 [2016-08-15 12:39:47,040] - [rest_client:2217] INFO - {"params": {"mapping": {"default_analyzer": "customAnalyzer1", "default_datetime_parser": "dateTimeOptional", "type_field": "type", "default_mapping": {"default_analyzer": "", "dynamic": true, "enabled": false}, "default_type": "_default", "analysis": {"analyzers": {"customAnalyzer1": {"type": "custom", "char_filters": ["html"], "tokenizer": "whitespace", "token_filters": ["shingle"]}}, "token_filters": {"dict_compound_en": {"dict_token_map": "stop_en", "type": "dict_compound"}, "length": {"max": 5, "type": "length", "min": 3}, "truncate": {"length": 10, "type": "truncate_token"}, "dict_compound_fr": {"dict_token_map": "articles_fr", "type": "dict_compound"}, "ngram": {"max": 5, "type": "ngram", "min": 3}, "front_edge_ngram": {"max": 5, "type": "edge_ngram", "back": false, "min": 3}, "back_edge_ngram": {"max": 5, "type": "edge_ngram", "back": true, "min": 3}, "shingle": {"output_original": "false", "min": 2, "filler": "", "max": 5, "separator": "", "type": "shingle"}, "keyword_marker_en": {"type": "keyword_marker", "keywords_token_map": "stop_en"}}}, "default_field": "_all", "types": {"emp": {"enabled": true, "dynamic": false, "properties": {"salary": {"enabled": true, "dynamic": false, "properties": {}, "fields": [{"index": true, "name": "salary", "include_in_all": true, "type": "number", "analyzer": "", "include_term_vectors": true, "store": false}]}, "dept": {"enabled": true, "dynamic": false, "properties": {}, "fields": [{"index": true, "name": "dept", "include_in_all": true, "type": "text", "analyzer": "customAnalyzer1", "include_term_vectors": true, "store": false}]}, "join_date": {"enabled": true, "dynamic": false, "properties": {}, "fields": [{"index": true, "name": "join_date", "include_in_all": true, "type": "datetime", "analyzer": "", "include_term_vectors": true, "store": false}]}, "name": {"enabled": true, "dynamic": false, "properties": {}, "fields": [{"index": true, "name": "name", "include_in_all": true, "type": "text", "analyzer": "customAnalyzer1", "include_term_vectors": true, "store": false}]}, "mutated": {"enabled": true, "dynamic": false, "properties": {}, "fields": [{"index": true, "name": "mutated", "include_in_all": true, "type": "number", "analyzer": "", "include_term_vectors": true, "store": false}]}}, "fields": []}}}}, "name": "custom_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-08-15 12:39:47,119] - [rest_client:2226] INFO - Index custom_index created [2016-08-15 12:39:47,119] - [fts_base:2584] INFO - Validating index distribution for custom_index ... [2016-08-15 12:39:47,147] - [fts_base:2495] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-08-15 12:39:52,297] - [fts_base:2602] INFO - Validated: Number of PIndexes = 32 [2016-08-15 12:39:52,312] - [fts_base:2614] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-08-15 12:39:52,313] - [fts_base:2630] INFO - Validated: pIndexes are distributed across [u'dcec109232afc5fdd72f56d81f17cc5c'] [2016-08-15 12:39:52,313] - [fts_base:2636] INFO - Expecting num of partitions in each node in range 992-1024 [2016-08-15 12:39:52,313] - [fts_base:2651] INFO - Validated: Node dcec109232afc5fdd72f56d81f17cc5c houses 32 pindexes which serve 1024 partitions [2016-08-15 12:39:52,341] - [es_base:260] INFO - Creating es_index with mapping { "mappings": { "emp": { "dynamic": false, "properties": { "salary": { "index": "analyzed", "type": "float", "store": false }, "dept": { "index": "analyzed", "type": "string", "analyzer": "customAnalyzer1", "store": false, "term_vector": "yes" }, "join_date": { "index": "analyzed", "type": "date", "store": false }, "name": { "index": "analyzed", "type": "string", "analyzer": "customAnalyzer1", "store": false, "term_vector": "yes" }, "mutated": { "index": "analyzed", "type": "float", "store": false } } } }, "settings": { "analysis": { "filter": { "truncate": { "length": 10, "type": "truncate" }, "shingle": { "token_separator": "", "output_unigrams": "false", "filler_token": "", "output_unigrams_if_no_shingles": "false", "min_shingle_size": 2, "type": "shingle", "max_shingle_size": 5 }, "ngram": { "max_gram": 5, "type": "nGram", "side": "front", "min_gram": 3 }, "keyword_marker_en": { "keywords": [ "i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "would", "should", "could", "ought", "i'm", "you're", "he's", "she's", "it's", "we're", "they're", "i've", "you've", "we've", "they've", "i'd", "you'd", "he'd", "she'd", "we'd", "they'd", "i'll", "you'll", "he'll", "she'll", "we'll", "they'll", "isn't", "aren't", "wasn't", "weren't", "hasn't", "haven't", "hadn't", "doesn't", "don't", "didn't", "won't", "wouldn't", "shan't", "shouldn't", "can't", "cannot", "couldn't", "mustn't", "let's", "that's", "who's", "what's", "here's", "there's", "when's", "where's", "why's", "how's", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very" ], "type": "keyword_marker" }, "length": { "max": 5, "type": "length", "min": 3 }, "front_edge_ngram": { "max_gram": 5, "type": "edgeNGram", "side": "front", "min_gram": 3 }, "back_edge_ngram": { "max_gram": 5, "type": "edgeNGram", "side": "back", "min_gram": 3 } }, "analyzer": { "customAnalyzer1": { "filter": [ "shingle" ], "char_filter": [ "html_strip" ], "tokenizer": "whitespace" } } } } } [2016-08-15 12:39:52,393] - [es_base:266] INFO - SUCCESS: ES index created with above mapping [2016-08-15 12:39:52,545] - [task:1044] INFO - Starting operation 'create' on Elastic Search ... [2016-08-15 12:39:52,686] - [data_helper:295] INFO - creating direct client 172.23.106.72:11210 default [2016-08-15 12:39:53,552] - [task:1078] INFO - 1000 documents bulk loaded into ES [2016-08-15 12:39:53,619] - [task:1083] INFO - ES index count for 'es_index': 1000 [2016-08-15 12:39:53,844] - [fts_base:2892] INFO - Loading phase complete! [2016-08-15 12:39:54,217] - [fts_base:2525] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 27, docs in ES index: 1000 [2016-08-15 12:40:00,360] - [fts_base:2525] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 572, docs in ES index: 1000 [2016-08-15 12:40:06,481] - [fts_base:2525] INFO - Docs in bucket = 1000, docs in FTS index 'custom_index': 1000, docs in ES index: 1000 [2016-08-15 12:40:06,482] - [fts_base:2537] INFO - FTS indexed 1000 docs in 0.21 mins Smart queries will be generated on fields: {'date': ['join_date'], 'text': ['name', 'dept'], 'num': ['salary', 'mutated'], 'str': ['name', 'dept']} [2016-08-15 12:40:07,223] - [task:1112] INFO - ------------------------------------------------------------------ Query # 1 ----------------------------------------------------------------- [2016-08-15 12:40:07,238] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "name", "match": "Trista Stewart"}, {"field": "name", "prefix": "D"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,290] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,290] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Trista Stewart"}, {"field": "name", "prefix": "D"}]} is 0 (took 31.797227ms) [2016-08-15 12:40:07,294] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"name": "Trista Stewart"}}, {"prefix": {"name": "D"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:07,295] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:07,295] - [task:1112] INFO - ------------------------------------------------------------------ Query # 2 ----------------------------------------------------------------- [2016-08-15 12:40:07,311] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Support"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,337] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,337] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Support"} is 0 (took 2.867092ms) [2016-08-15 12:40:07,342] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Support"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:07,343] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:07,343] - [task:1112] INFO - ------------------------------------------------------------------ Query # 3 ----------------------------------------------------------------- [2016-08-15 12:40:07,362] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Dev-ops"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,394] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,394] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Dev-ops"} is 0 (took 3.914214ms) [2016-08-15 12:40:07,398] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Dev-ops"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:07,398] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:07,398] - [task:1112] INFO - ------------------------------------------------------------------ Query # 4 ----------------------------------------------------------------- [2016-08-15 12:40:07,417] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "prefix": "Pr"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,453] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,454] - [task:1140] INFO - FTS hits for query: {"field": "dept", "prefix": "Pr"} is 99 (took 5.217532ms) [2016-08-15 12:40:07,457] - [task:1150] INFO - ES hits for query: {"prefix": {"dept": "Pr"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:07,458] - [task:1155] ERROR - FAIL: FTS hits: 99, while ES hits: 0 [2016-08-15 12:40:07,458] - [task:1170] ERROR - FAIL: Following 99 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000732', u'emp10000124', u'emp10000231', u'emp10000230', u'emp10000438', u'emp10000877', u'emp10000871', u'emp10000703', u'emp10000331', u'emp10000426', u'emp10000548', u'emp10000420', u'emp10000542', u'emp10000541', u'emp10000038', u'emp10000584', u'emp10000984', u'emp10000630', u'emp10000903', u'emp10000653', u'emp10000305', u'emp10000655', u'emp10000145', u'emp10000143', u'emp10000414', u'emp10000416', u'emp10000021', u'emp10000791', u'emp10000025', u'emp10000251', u'emp10000250', u'emp10000798', u'emp10000994', u'emp10000625', u'emp10000289', u'emp10000640', u'emp10000728', u'emp10000648', u'emp10000405', u'emp10000017', u'emp10000016', u'emp10000568', u'emp10000399', u'emp10000565', u'emp10000561', u'emp10000847', u'emp10000719', u'emp10000496', u'emp10000499', u'emp10000710'] [2016-08-15 12:40:07,458] - [task:1112] INFO - ------------------------------------------------------------------ Query # 5 ----------------------------------------------------------------- [2016-08-15 12:40:07,474] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 120131.0, "inclusive_min": true, "min": 111000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,516] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,517] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 120131.0, "inclusive_min": true, "min": 111000.0} is 84 (took 18.358993ms) [2016-08-15 12:40:07,527] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"gte": 111000.0, "lte": 120131.0}}}}} on es_index is 84 (took 2ms) [2016-08-15 12:40:07,528] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:07,528] - [task:1112] INFO - ------------------------------------------------------------------ Query # 6 ----------------------------------------------------------------- [2016-08-15 12:40:07,545] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "wildcard": "Finan?e"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,578] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,578] - [task:1140] INFO - FTS hits for query: {"field": "dept", "wildcard": "Finan?e"} is 0 (took 4.154431ms) [2016-08-15 12:40:07,582] - [task:1150] INFO - ES hits for query: {"wildcard": {"dept": "Finan?e"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:07,582] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:07,582] - [task:1112] INFO - ------------------------------------------------------------------ Query # 7 ----------------------------------------------------------------- [2016-08-15 12:40:07,600] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Richardson*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,629] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,629] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Richardson*"} is 20 (took 4.505438ms) [2016-08-15 12:40:07,634] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Richardson*"}} on es_index is 11 (took 1ms) [2016-08-15 12:40:07,635] - [task:1155] ERROR - FAIL: FTS hits: 20, while ES hits: 11 [2016-08-15 12:40:07,635] - [task:1170] ERROR - FAIL: Following 9 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000083', u'emp10000673', u'emp10000725', u'emp10000147', u'emp10000133', u'emp10000623', u'emp10000776', u'emp10000203', u'emp10000375'] [2016-08-15 12:40:07,635] - [task:1112] INFO - ------------------------------------------------------------------ Query # 8 ----------------------------------------------------------------- [2016-08-15 12:40:07,654] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "mutated:>4 +mutated:<=2444 salary:<147000.0 +dept:\"Finance\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,789] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,789] - [task:1140] INFO - FTS hits for query: {"query": "mutated:>4 +mutated:<=2444 salary:<147000.0 +dept:\"Finance\""} is 0 (took 110.217102ms) [2016-08-15 12:40:07,928] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "mutated:>4 +mutated:<=2444 salary:<147000.0 +dept:\"Finance\""}} on es_index is 1000 (took 17ms) [2016-08-15 12:40:07,929] - [task:1155] ERROR - FAIL: FTS hits: 0, while ES hits: 1000 [2016-08-15 12:40:07,930] - [task:1170] ERROR - FAIL: Following 1000 docs were not returned by FTS, but ES, printing 50: [u'emp10000538', u'emp10000539', u'emp10000536', u'emp10000537', u'emp10000534', u'emp10000535', u'emp10000532', u'emp10000533', u'emp10000530', u'emp10000531', u'emp10000125', u'emp10000436', u'emp10000435', u'emp10000126', u'emp10000433', u'emp10000120', u'emp10000431', u'emp10000430', u'emp10000129', u'emp10000128', u'emp10000439', u'emp10000438', u'emp10000509', u'emp10000508', u'emp10000192', u'emp10000471', u'emp10000721', u'emp10000417', u'emp10000222', u'emp10000223', u'emp10000549', u'emp10000221', u'emp10000226', u'emp10000227', u'emp10000224', u'emp10000225', u'emp10000543', u'emp10000542', u'emp10000228', u'emp10000229', u'emp10000547', u'emp10000546', u'emp10000545', u'emp10000544', u'emp10000983', u'emp10000982', u'emp10000981', u'emp10000980', u'emp10000987', u'emp10000986'] [2016-08-15 12:40:07,930] - [task:1112] INFO - ------------------------------------------------------------------ Query # 9 ----------------------------------------------------------------- [2016-08-15 12:40:07,946] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "Quel"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:07,978] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:07,979] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "Quel"} is 27 (took 4.701327ms) [2016-08-15 12:40:07,985] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "Quel"}} on es_index is 14 (took 2ms) [2016-08-15 12:40:07,986] - [task:1155] ERROR - FAIL: FTS hits: 27, while ES hits: 14 [2016-08-15 12:40:07,986] - [task:1170] ERROR - FAIL: Following 13 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000314', u'emp10000993', u'emp10000412', u'emp10000091', u'emp10000527', u'emp10000864', u'emp10000283', u'emp10000364', u'emp10000817', u'emp10000759', u'emp10000554', u'emp10000602', u'emp10000557'] [2016-08-15 12:40:07,986] - [task:1112] INFO - ------------------------------------------------------------------ Query # 10 ----------------------------------------------------------------- [2016-08-15 12:40:07,998] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "+dept:\"Sales\" salary:>69000.0 +salary:<69331.0 dept:\"Dev-ops\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,086] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,086] - [task:1140] INFO - FTS hits for query: {"query": "+dept:\"Sales\" salary:>69000.0 +salary:<69331.0 dept:\"Dev-ops\""} is 0 (took 69.327075ms) [2016-08-15 12:40:08,105] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "+dept:\"Sales\" salary:>69000.0 +salary:<69331.0 dept:\"Dev-ops\""}} on es_index is 203 (took 4ms) [2016-08-15 12:40:08,105] - [task:1155] ERROR - FAIL: FTS hits: 0, while ES hits: 203 [2016-08-15 12:40:08,106] - [task:1170] ERROR - FAIL: Following 203 docs were not returned by FTS, but ES, printing 50: [u'emp10000731', u'emp10000044', u'emp10000735', u'emp10000739', u'emp10000225', u'emp10000048', u'emp10000436', u'emp10000126', u'emp10000239', u'emp10000430', u'emp10000236', u'emp10000232', u'emp10000978', u'emp10000694', u'emp10000876', u'emp10000873', u'emp10000604', u'emp10000600', u'emp10000601', u'emp10000707', u'emp10000706', u'emp10000338', u'emp10000334', u'emp10000335', u'emp10000341', u'emp10000746', u'emp10000709', u'emp10000137', u'emp10000333', u'emp10000425', u'emp10000426', u'emp10000789', u'emp10000321', u'emp10000224', u'emp10000423', u'emp10000786', u'emp10000781', u'emp10000429', u'emp10000983', u'emp10000981', u'emp10000987', u'emp10001000', u'emp10000631', u'emp10000630', u'emp10000866', u'emp10000867', u'emp10000581', u'emp10000863', u'emp10000900', u'emp10000905'] [2016-08-15 12:40:08,106] - [task:1112] INFO - ------------------------------------------------------------------ Query # 11 ----------------------------------------------------------------- [2016-08-15 12:40:08,124] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Pre-sales"}, {"field": "dept", "match": "Finance"}, {"field": "dept", "match": "Support"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,153] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,153] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Pre-sales"}, {"field": "dept", "match": "Finance"}, {"field": "dept", "match": "Support"}]} is 9 (took 6.172896ms) [2016-08-15 12:40:08,157] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Pre-sales"}}, {"match": {"dept": "Finance"}}, {"match": {"dept": "Support"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:08,157] - [task:1155] ERROR - FAIL: FTS hits: 9, while ES hits: 0 [2016-08-15 12:40:08,157] - [task:1170] ERROR - FAIL: Following 9 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000894', u'emp10000981', u'emp10000679', u'emp10000141', u'emp10000526', u'emp10000022', u'emp10000374', u'emp10000027', u'emp10000606'] [2016-08-15 12:40:08,158] - [task:1112] INFO - ------------------------------------------------------------------ Query # 12 ----------------------------------------------------------------- [2016-08-15 12:40:08,170] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.486108", "start": "2005-06-08T11:38:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,275] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,276] - [task:1140] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.486108", "start": "2005-06-08T11:38:00"} is 165 (took 74.607798ms) [2016-08-15 12:40:08,292] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"gte": "2005-06-08T11:38:00", "lte": "2016-08-15T12:40:06.486108"}}}}} on es_index is 165 (took 4ms) [2016-08-15 12:40:08,293] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:08,293] - [task:1112] INFO - ------------------------------------------------------------------ Query # 13 ----------------------------------------------------------------- [2016-08-15 12:40:08,305] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match": "Casondrah"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,328] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,328] - [task:1140] INFO - FTS hits for query: {"field": "name", "match": "Casondrah"} is 1 (took 2.914095ms) [2016-08-15 12:40:08,332] - [task:1150] INFO - ES hits for query: {"match": {"name": "Casondrah"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:08,332] - [task:1155] ERROR - FAIL: FTS hits: 1, while ES hits: 0 [2016-08-15 12:40:08,332] - [task:1170] ERROR - FAIL: Following 1 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000217'] [2016-08-15 12:40:08,333] - [task:1112] INFO - ------------------------------------------------------------------ Query # 14 ----------------------------------------------------------------- [2016-08-15 12:40:08,349] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "wildcard": "Suppo?t"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,374] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,374] - [task:1140] INFO - FTS hits for query: {"field": "dept", "wildcard": "Suppo?t"} is 0 (took 3.260986ms) [2016-08-15 12:40:08,379] - [task:1150] INFO - ES hits for query: {"wildcard": {"dept": "Suppo?t"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:08,379] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:08,380] - [task:1112] INFO - ------------------------------------------------------------------ Query # 15 ----------------------------------------------------------------- [2016-08-15 12:40:08,397] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 132327.0, "inclusive_min": true, "min": 129000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,460] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,461] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 132327.0, "inclusive_min": true, "min": 129000.0} is 33 (took 36.072646ms) [2016-08-15 12:40:08,467] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"gte": 129000.0, "lte": 132327.0}}}}} on es_index is 33 (took 2ms) [2016-08-15 12:40:08,468] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:08,468] - [task:1112] INFO - ------------------------------------------------------------------ Query # 16 ----------------------------------------------------------------- [2016-08-15 12:40:08,487] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "dept", "match": "HR"}, {"field": "dept", "match": "HR"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,522] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,522] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "dept", "match": "HR"}, {"field": "dept", "match": "HR"}]} is 4 (took 5.945509ms) [2016-08-15 12:40:08,526] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Support"}}, {"match": {"dept": "HR"}}, {"match": {"dept": "HR"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:08,526] - [task:1155] ERROR - FAIL: FTS hits: 4, while ES hits: 0 [2016-08-15 12:40:08,527] - [task:1170] ERROR - FAIL: Following 4 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000860', u'emp10000569', u'emp10000007', u'emp10000673'] [2016-08-15 12:40:08,527] - [task:1112] INFO - ------------------------------------------------------------------ Query # 17 ----------------------------------------------------------------- [2016-08-15 12:40:08,542] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "White"}, {"field": "name", "match": "Safiya Taylor"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,566] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,566] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "White"}, {"field": "name", "match": "Safiya Taylor"}]} is 0 (took 5.470701ms) [2016-08-15 12:40:08,579] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "White"}}, {"match": {"name": "Safiya Taylor"}}]}} on es_index is 0 (took 9ms) [2016-08-15 12:40:08,580] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:08,580] - [task:1112] INFO - ------------------------------------------------------------------ Query # 18 ----------------------------------------------------------------- [2016-08-15 12:40:08,591] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "-mutated:>1 +mutated:<4794 mutated:>5 +mutated:<2215 -dept:\"HR\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:08,892] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:08,893] - [task:1140] INFO - FTS hits for query: {"query": "-mutated:>1 +mutated:<4794 mutated:>5 +mutated:<2215 -dept:\"HR\""} is 1000 (took 243.120321ms) [2016-08-15 12:40:08,974] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "-mutated:>1 +mutated:<4794 mutated:>5 +mutated:<2215 -dept:\"HR\""}} on es_index is 1000 (took 16ms) [2016-08-15 12:40:08,976] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:08,977] - [task:1112] INFO - ------------------------------------------------------------------ Query # 19 ----------------------------------------------------------------- [2016-08-15 12:40:08,991] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 131384.0, "inclusive_min": true, "min": 123000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:09,032] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:09,033] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 131384.0, "inclusive_min": true, "min": 123000.0} is 85 (took 17.151148ms) [2016-08-15 12:40:09,043] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"gte": 123000.0, "lte": 131384.0}}}}} on es_index is 85 (took 3ms) [2016-08-15 12:40:09,043] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:09,044] - [task:1112] INFO - ------------------------------------------------------------------ Query # 20 ----------------------------------------------------------------- [2016-08-15 12:40:09,064] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Tr*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:09,123] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:09,123] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Tr*"} is 203 (took 17.395388ms) [2016-08-15 12:40:09,130] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Tr*"}} on es_index is 39 (took 2ms) [2016-08-15 12:40:09,131] - [task:1155] ERROR - FAIL: FTS hits: 203, while ES hits: 39 [2016-08-15 12:40:09,131] - [task:1170] ERROR - FAIL: Following 164 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000046', u'emp10000043', u'emp10000040', u'emp10000326', u'emp10000323', u'emp10000320', u'emp10000239', u'emp10000231', u'emp10000439', u'emp10000329', u'emp10000743', u'emp10000877', u'emp10000604', u'emp10000707', u'emp10000338', u'emp10000702', u'emp10000334', u'emp10000332', u'emp10000135', u'emp10000031', u'emp10000030', u'emp10000037', u'emp10000049', u'emp10000547', u'emp10000816', u'emp10001000', u'emp10000866', u'emp10000360', u'emp10000635', u'emp10000861', u'emp10000639', u'emp10000907', u'emp10000101', u'emp10000419', u'emp10000300', u'emp10000650', u'emp10000656', u'emp10000654', u'emp10000308', u'emp10000659', u'emp10000658', u'emp10000559', u'emp10000790', u'emp10000797', u'emp10000029', u'emp10000552', u'emp10000387', u'emp10000258', u'emp10000269', u'emp10000850'] [2016-08-15 12:40:09,131] - [task:1112] INFO - ------------------------------------------------------------------ Query # 21 ----------------------------------------------------------------- [2016-08-15 12:40:09,145] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 9215, "inclusive_min": true, "min": 0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:09,399] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:09,399] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 9215, "inclusive_min": true, "min": 0} is 1000 (took 24.58499ms) [2016-08-15 12:40:09,469] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"gte": 0, "lte": 9215}}}}} on es_index is 1000 (took 18ms) [2016-08-15 12:40:09,471] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:09,472] - [task:1112] INFO - ------------------------------------------------------------------ Query # 22 ----------------------------------------------------------------- [2016-08-15 12:40:09,490] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 96692.0, "inclusive_min": true, "min": 91000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:09,526] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:09,527] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 96692.0, "inclusive_min": true, "min": 91000.0} is 65 (took 12.298925ms) [2016-08-15 12:40:09,535] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"gte": 91000.0, "lte": 96692.0}}}}} on es_index is 65 (took 2ms) [2016-08-15 12:40:09,535] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:09,536] - [task:1112] INFO - ------------------------------------------------------------------ Query # 23 ----------------------------------------------------------------- [2016-08-15 12:40:09,551] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match": "Gallia"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:09,577] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:09,577] - [task:1140] INFO - FTS hits for query: {"field": "name", "match": "Gallia"} is 1 (took 3.946905ms) [2016-08-15 12:40:09,581] - [task:1150] INFO - ES hits for query: {"match": {"name": "Gallia"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:09,581] - [task:1155] ERROR - FAIL: FTS hits: 1, while ES hits: 0 [2016-08-15 12:40:09,581] - [task:1170] ERROR - FAIL: Following 1 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000788'] [2016-08-15 12:40:09,582] - [task:1112] INFO - ------------------------------------------------------------------ Query # 24 ----------------------------------------------------------------- [2016-08-15 12:40:09,596] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,062] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,062] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "*"} is 1000 (took 286.430565ms) [2016-08-15 12:40:10,136] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "*"}} on es_index is 1000 (took 17ms) [2016-08-15 12:40:10,138] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,138] - [task:1112] INFO - ------------------------------------------------------------------ Query # 25 ----------------------------------------------------------------- [2016-08-15 12:40:10,151] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.489832", "start": "2013-05-09T08:56:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,222] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,223] - [task:1140] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.489832", "start": "2013-05-09T08:56:00"} is 46 (took 46.905411ms) [2016-08-15 12:40:10,230] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"gte": "2013-05-09T08:56:00", "lte": "2016-08-15T12:40:06.489832"}}}}} on es_index is 46 (took 1ms) [2016-08-15 12:40:10,230] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,230] - [task:1112] INFO - ------------------------------------------------------------------ Query # 26 ----------------------------------------------------------------- [2016-08-15 12:40:10,257] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "name", "match": "Riona"}, {"field": "dept", "prefix": "P"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.490336", "start": "1997-09-25T00:52:00"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,325] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,326] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Riona"}, {"field": "dept", "prefix": "P"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.490336", "start": "1997-09-25T00:52:00"}]} is 0 (took 42.783368ms) [2016-08-15 12:40:10,330] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"name": "Riona"}}, {"prefix": {"dept": "P"}}, {"filtered": {"filter": {"range": {"join_date": {"gte": "1997-09-25T00:52:00", "lte": "2016-08-15T12:40:06.490336"}}}}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:10,331] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,331] - [task:1112] INFO - ------------------------------------------------------------------ Query # 27 ----------------------------------------------------------------- [2016-08-15 12:40:10,345] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Sales"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,377] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,378] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Sales"} is 0 (took 4.405227ms) [2016-08-15 12:40:10,381] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Sales"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:10,381] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,382] - [task:1112] INFO - ------------------------------------------------------------------ Query # 28 ----------------------------------------------------------------- [2016-08-15 12:40:10,398] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Trista"}, {"field": "dept", "match": "Accounts"}, {"field": "dept", "match": "Finance"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,452] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,453] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Trista"}, {"field": "dept", "match": "Accounts"}, {"field": "dept", "match": "Finance"}]} is 2 (took 30.357415ms) [2016-08-15 12:40:10,457] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "Trista"}}, {"match": {"dept": "Accounts"}}, {"match": {"dept": "Finance"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:10,457] - [task:1155] ERROR - FAIL: FTS hits: 2, while ES hits: 0 [2016-08-15 12:40:10,458] - [task:1170] ERROR - FAIL: Following 2 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000565', u'emp10000434'] [2016-08-15 12:40:10,458] - [task:1112] INFO - ------------------------------------------------------------------ Query # 29 ----------------------------------------------------------------- [2016-08-15 12:40:10,474] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "prefix": "Dev-"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,509] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,509] - [task:1140] INFO - FTS hits for query: {"field": "dept", "prefix": "Dev-"} is 83 (took 6.754296ms) [2016-08-15 12:40:10,512] - [task:1150] INFO - ES hits for query: {"prefix": {"dept": "Dev-"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:10,513] - [task:1155] ERROR - FAIL: FTS hits: 83, while ES hits: 0 [2016-08-15 12:40:10,513] - [task:1170] ERROR - FAIL: Following 83 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000047', u'emp10000303', u'emp10000162', u'emp10000657', u'emp10000860', u'emp10000736', u'emp10000292', u'emp10000002', u'emp10000739', u'emp10000272', u'emp10000679', u'emp10000414', u'emp10000345', u'emp10000388', u'emp10000435', u'emp10000576', u'emp10000511', u'emp10000104', u'emp10000028', u'emp10000943', u'emp10000382', u'emp10000450', u'emp10000215', u'emp10000386', u'emp10000408', u'emp10000996', u'emp10000850', u'emp10000306', u'emp10000107', u'emp10000970', u'emp10000184', u'emp10000874', u'emp10000839', u'emp10000742', u'emp10000609', u'emp10000480', u'emp10000870', u'emp10000663', u'emp10000938', u'emp10000474', u'emp10000722', u'emp10000706', u'emp10000158', u'emp10000497', u'emp10000057', u'emp10000075', u'emp10000953', u'emp10000262', u'emp10000137', u'emp10000318'] [2016-08-15 12:40:10,513] - [task:1112] INFO - ------------------------------------------------------------------ Query # 30 ----------------------------------------------------------------- [2016-08-15 12:40:10,533] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 839, "inclusive_min": true, "min": 3}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,575] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,575] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 839, "inclusive_min": true, "min": 3} is 0 (took 15.412493ms) [2016-08-15 12:40:10,579] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"gte": 3, "lte": 839}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:10,580] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,580] - [task:1112] INFO - ------------------------------------------------------------------ Query # 31 ----------------------------------------------------------------- [2016-08-15 12:40:10,597] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "salary", "max": 149176.0, "inclusive_min": false, "min": 145000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,678] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,678] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "salary", "max": 149176.0, "inclusive_min": false, "min": 145000.0} is 39 (took 55.011976ms) [2016-08-15 12:40:10,686] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"lt": 149176.0, "gt": 145000.0}}}}} on es_index is 39 (took 2ms) [2016-08-15 12:40:10,687] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,687] - [task:1112] INFO - ------------------------------------------------------------------ Query # 32 ----------------------------------------------------------------- [2016-08-15 12:40:10,703] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.492187", "start": "1950-10-15T02:07:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:10,837] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:10,838] - [task:1140] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.492187", "start": "1950-10-15T02:07:00"} is 978 (took 64.723458ms) [2016-08-15 12:40:10,916] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"gte": "1950-10-15T02:07:00", "lte": "2016-08-15T12:40:06.492187"}}}}} on es_index is 978 (took 17ms) [2016-08-15 12:40:10,918] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:10,918] - [task:1112] INFO - ------------------------------------------------------------------ Query # 33 ----------------------------------------------------------------- [2016-08-15 12:40:10,944] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "salary", "max": 120824.0, "inclusive_min": false, "min": 119000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,020] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,020] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "salary", "max": 120824.0, "inclusive_min": false, "min": 119000.0} is 17 (took 43.11038ms) [2016-08-15 12:40:11,026] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"lt": 120824.0, "gt": 119000.0}}}}} on es_index is 17 (took 1ms) [2016-08-15 12:40:11,026] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,026] - [task:1112] INFO - ------------------------------------------------------------------ Query # 34 ----------------------------------------------------------------- [2016-08-15 12:40:11,044] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "name", "match": "Solita Mitchell"}, {"field": "name", "match_phrase": "Kerry-Ann Simón"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.492973", "start": "2008-04-01T00:40:00"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,143] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,143] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Solita Mitchell"}, {"field": "name", "match_phrase": "Kerry-Ann Simón"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.492973", "start": "2008-04-01T00:40:00"}]} is 0 (took 66.108156ms) [2016-08-15 12:40:11,148] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"name": "Solita Mitchell"}}, {"match_phrase": {"name": "Kerry-Ann Simón"}}, {"filtered": {"filter": {"range": {"join_date": {"gte": "2008-04-01T00:40:00", "lte": "2016-08-15T12:40:06.492973"}}}}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,148] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,148] - [task:1112] INFO - ------------------------------------------------------------------ Query # 35 ----------------------------------------------------------------- [2016-08-15 12:40:11,166] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Engineering"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,195] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,196] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Engineering"} is 1 (took 4.785942ms) [2016-08-15 12:40:11,200] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Engineering"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,201] - [task:1155] ERROR - FAIL: FTS hits: 1, while ES hits: 0 [2016-08-15 12:40:11,201] - [task:1170] ERROR - FAIL: Following 1 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000436'] [2016-08-15 12:40:11,201] - [task:1112] INFO - ------------------------------------------------------------------ Query # 36 ----------------------------------------------------------------- [2016-08-15 12:40:11,220] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Pre-sales"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,253] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,254] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Pre-sales"} is 0 (took 5.331044ms) [2016-08-15 12:40:11,258] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Pre-sales"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,259] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,259] - [task:1112] INFO - ------------------------------------------------------------------ Query # 37 ----------------------------------------------------------------- [2016-08-15 12:40:11,275] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Pre-sales"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,309] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,310] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Pre-sales"} is 0 (took 4.308145ms) [2016-08-15 12:40:11,314] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Pre-sales"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,314] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,314] - [task:1112] INFO - ------------------------------------------------------------------ Query # 38 ----------------------------------------------------------------- [2016-08-15 12:40:11,328] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "K"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,396] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,397] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "K"} is 289 (took 19.11534ms) [2016-08-15 12:40:11,414] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "K"}} on es_index is 161 (took 4ms) [2016-08-15 12:40:11,414] - [task:1155] ERROR - FAIL: FTS hits: 289, while ES hits: 161 [2016-08-15 12:40:11,415] - [task:1170] ERROR - FAIL: Following 128 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000046', u'emp10000596', u'emp10000044', u'emp10000737', u'emp10000739', u'emp10000048', u'emp10000125', u'emp10000543', u'emp10000127', u'emp10000434', u'emp10000432', u'emp10000122', u'emp10000231', u'emp10000232', u'emp10000872', u'emp10000600', u'emp10000138', u'emp10000706', u'emp10000335', u'emp10000130', u'emp10000131', u'emp10000134', u'emp10000333', u'emp10000549', u'emp10000035', u'emp10000785', u'emp10000428', u'emp10000517', u'emp10000544', u'emp10001000', u'emp10000867', u'emp10000635', u'emp10000903', u'emp10000638', u'emp10000905', u'emp10000657', u'emp10000148', u'emp10000309', u'emp10000417', u'emp10000388', u'emp10000720', u'emp10000513', u'emp10000251', u'emp10000027', u'emp10000551', u'emp10000798', u'emp10000384', u'emp10000990', u'emp10000855', u'emp10000521'] [2016-08-15 12:40:11,415] - [task:1112] INFO - ------------------------------------------------------------------ Query # 39 ----------------------------------------------------------------- [2016-08-15 12:40:11,430] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.494391", "start": "1974-02-22T00:16:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,548] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,549] - [task:1140] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.494391", "start": "1974-02-22T00:16:00"} is 633 (took 56.320012ms) [2016-08-15 12:40:11,598] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"lt": "2016-08-15T12:40:06.494391", "gt": "1974-02-22T00:16:00"}}}}} on es_index is 633 (took 15ms) [2016-08-15 12:40:11,599] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,599] - [task:1112] INFO - ------------------------------------------------------------------ Query # 40 ----------------------------------------------------------------- [2016-08-15 12:40:11,615] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match": "Ciara"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,647] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,647] - [task:1140] INFO - FTS hits for query: {"field": "name", "match": "Ciara"} is 2 (took 3.753554ms) [2016-08-15 12:40:11,651] - [task:1150] INFO - ES hits for query: {"match": {"name": "Ciara"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,651] - [task:1155] ERROR - FAIL: FTS hits: 2, while ES hits: 0 [2016-08-15 12:40:11,652] - [task:1170] ERROR - FAIL: Following 2 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000679', u'emp10000779'] [2016-08-15 12:40:11,652] - [task:1112] INFO - ------------------------------------------------------------------ Query # 41 ----------------------------------------------------------------- [2016-08-15 12:40:11,668] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 3085, "inclusive_min": true, "min": 5}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,728] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,729] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 3085, "inclusive_min": true, "min": 5} is 0 (took 28.112732ms) [2016-08-15 12:40:11,733] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"gte": 5, "lte": 3085}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,734] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,734] - [task:1112] INFO - ------------------------------------------------------------------ Query # 42 ----------------------------------------------------------------- [2016-08-15 12:40:11,747] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "S?fiy? Anderson"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,779] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,780] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "S?fiy? Anderson"} is 0 (took 9.22898ms) [2016-08-15 12:40:11,783] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "S?fiy? Anderson"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,783] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,784] - [task:1112] INFO - ------------------------------------------------------------------ Query # 43 ----------------------------------------------------------------- [2016-08-15 12:40:11,804] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Training"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,832] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,833] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Training"} is 0 (took 4.198597ms) [2016-08-15 12:40:11,838] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Training"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,838] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,839] - [task:1112] INFO - ------------------------------------------------------------------ Query # 44 ----------------------------------------------------------------- [2016-08-15 12:40:11,853] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "prefix": "In"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,892] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,892] - [task:1140] INFO - FTS hits for query: {"field": "dept", "prefix": "In"} is 85 (took 5.730646ms) [2016-08-15 12:40:11,896] - [task:1150] INFO - ES hits for query: {"prefix": {"dept": "In"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,896] - [task:1155] ERROR - FAIL: FTS hits: 85, while ES hits: 0 [2016-08-15 12:40:11,896] - [task:1170] ERROR - FAIL: Following 85 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000675', u'emp10000730', u'emp10000162', u'emp10000304', u'emp10000041', u'emp10000271', u'emp10000659', u'emp10000272', u'emp10000415', u'emp10000061', u'emp10000964', u'emp10000103', u'emp10000573', u'emp10000361', u'emp10000257', u'emp10000024', u'emp10000366', u'emp10000026', u'emp10000457', u'emp10000453', u'emp10000232', u'emp10000686', u'emp10000463', u'emp10000891', u'emp10000995', u'emp10000117', u'emp10000512', u'emp10000850', u'emp10000974', u'emp10000852', u'emp10000876', u'emp10000801', u'emp10000770', u'emp10000818', u'emp10000526', u'emp10000662', u'emp10000605', u'emp10000660', u'emp10000607', u'emp10000557', u'emp10000946', u'emp10000714', u'emp10000174', u'emp10000313', u'emp10000268', u'emp10000598', u'emp10000132', u'emp10000335', u'emp10000130', u'emp10000351'] [2016-08-15 12:40:11,897] - [task:1112] INFO - ------------------------------------------------------------------ Query # 45 ----------------------------------------------------------------- [2016-08-15 12:40:11,910] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Training"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:11,936] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:11,937] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Training"} is 0 (took 2.526859ms) [2016-08-15 12:40:11,940] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Training"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:11,940] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:11,941] - [task:1112] INFO - ------------------------------------------------------------------ Query # 46 ----------------------------------------------------------------- [2016-08-15 12:40:11,956] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "mutated:>5 +mutated:<=665"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,109] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,110] - [task:1140] INFO - FTS hits for query: {"query": "mutated:>5 +mutated:<=665"} is 1000 (took 79.144365ms) [2016-08-15 12:40:12,184] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "mutated:>5 +mutated:<=665"}} on es_index is 1000 (took 22ms) [2016-08-15 12:40:12,185] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,186] - [task:1112] INFO - ------------------------------------------------------------------ Query # 47 ----------------------------------------------------------------- [2016-08-15 12:40:12,206] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Trista"}, {"field": "dept", "match": "Pre-sales"}, {"field": "name", "match": "Cassandra Sr."}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,239] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,239] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Trista"}, {"field": "dept", "match": "Pre-sales"}, {"field": "name", "match": "Cassandra Sr."}]} is 3 (took 8.079254ms) [2016-08-15 12:40:12,243] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "Trista"}}, {"match": {"dept": "Pre-sales"}}, {"match": {"name": "Cassandra Sr."}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,244] - [task:1155] ERROR - FAIL: FTS hits: 3, while ES hits: 0 [2016-08-15 12:40:12,244] - [task:1170] ERROR - FAIL: Following 3 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000522', u'emp10000193', u'emp10000747'] [2016-08-15 12:40:12,244] - [task:1112] INFO - ------------------------------------------------------------------ Query # 48 ----------------------------------------------------------------- [2016-08-15 12:40:12,258] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Delores Willi*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,287] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,287] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Delores Willi*"} is 0 (took 4.2329ms) [2016-08-15 12:40:12,290] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Delores Willi*"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,291] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,291] - [task:1112] INFO - ------------------------------------------------------------------ Query # 49 ----------------------------------------------------------------- [2016-08-15 12:40:12,306] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "name", "match": "Caryssa Johnson"}, {"field": "dept", "match": "Support"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,342] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,343] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "name", "match": "Caryssa Johnson"}, {"field": "dept", "match": "Support"}]} is 4 (took 7.806241ms) [2016-08-15 12:40:12,347] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Support"}}, {"match": {"name": "Caryssa Johnson"}}, {"match": {"dept": "Support"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,348] - [task:1155] ERROR - FAIL: FTS hits: 4, while ES hits: 0 [2016-08-15 12:40:12,348] - [task:1170] ERROR - FAIL: Following 4 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000832', u'emp10000974', u'emp10000603', u'emp10000724'] [2016-08-15 12:40:12,349] - [task:1112] INFO - ------------------------------------------------------------------ Query # 50 ----------------------------------------------------------------- [2016-08-15 12:40:12,366] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "salary", "max": 93465.0, "inclusive_min": false, "min": 88000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,473] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,473] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "salary", "max": 93465.0, "inclusive_min": false, "min": 88000.0} is 53 (took 78.211813ms) [2016-08-15 12:40:12,481] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"lt": 93465.0, "gt": 88000.0}}}}} on es_index is 53 (took 1ms) [2016-08-15 12:40:12,482] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,482] - [task:1112] INFO - ------------------------------------------------------------------ Query # 51 ----------------------------------------------------------------- [2016-08-15 12:40:12,499] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 5642, "inclusive_min": true, "min": 2}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,537] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,538] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 5642, "inclusive_min": true, "min": 2} is 0 (took 10.1585ms) [2016-08-15 12:40:12,541] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"gte": 2, "lte": 5642}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,542] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,542] - [task:1112] INFO - ------------------------------------------------------------------ Query # 52 ----------------------------------------------------------------- [2016-08-15 12:40:12,556] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "-name:\"Josué\" mutated:<5 mutated:>4 +mutated:<8186"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,776] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,777] - [task:1140] INFO - FTS hits for query: {"query": "-name:\"Josué\" mutated:<5 mutated:>4 +mutated:<8186"} is 1000 (took 144.289488ms) [2016-08-15 12:40:12,853] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "-name:\"Josué\" mutated:<5 mutated:>4 +mutated:<8186"}} on es_index is 1000 (took 17ms) [2016-08-15 12:40:12,855] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,856] - [task:1112] INFO - ------------------------------------------------------------------ Query # 53 ----------------------------------------------------------------- [2016-08-15 12:40:12,877] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "HR"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,910] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,911] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "HR"} is 0 (took 6.230287ms) [2016-08-15 12:40:12,915] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "HR"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,916] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,916] - [task:1112] INFO - ------------------------------------------------------------------ Query # 54 ----------------------------------------------------------------- [2016-08-15 12:40:12,930] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Engineering"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:12,953] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:12,953] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Engineering"} is 0 (took 2.952511ms) [2016-08-15 12:40:12,956] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Engineering"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:12,957] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:12,957] - [task:1112] INFO - ------------------------------------------------------------------ Query # 55 ----------------------------------------------------------------- [2016-08-15 12:40:12,975] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Balandria Anderson"}, {"field": "dept", "match": "Info-tech"}, {"field": "dept", "match": "Accounts"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,006] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,007] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Balandria Anderson"}, {"field": "dept", "match": "Info-tech"}, {"field": "dept", "match": "Accounts"}]} is 5 (took 11.403632ms) [2016-08-15 12:40:13,010] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "Balandria Anderson"}}, {"match": {"dept": "Info-tech"}}, {"match": {"dept": "Accounts"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,011] - [task:1155] ERROR - FAIL: FTS hits: 5, while ES hits: 0 [2016-08-15 12:40:13,011] - [task:1170] ERROR - FAIL: Following 5 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000103', u'emp10000029', u'emp10000219', u'emp10000204', u'emp10000686'] [2016-08-15 12:40:13,011] - [task:1112] INFO - ------------------------------------------------------------------ Query # 56 ----------------------------------------------------------------- [2016-08-15 12:40:13,024] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "name", "match": "Sirena Scott"}, {"field": "dept", "prefix": "Fin"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.499612", "start": "2000-04-26T10:58:00"}, {"inclusive_max": true, "field": "mutated", "max": 2340, "inclusive_min": true, "min": 3}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,107] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,107] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Sirena Scott"}, {"field": "dept", "prefix": "Fin"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.499612", "start": "2000-04-26T10:58:00"}, {"inclusive_max": true, "field": "mutated", "max": 2340, "inclusive_min": true, "min": 3}]} is 0 (took 62.760913ms) [2016-08-15 12:40:13,111] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"name": "Sirena Scott"}}, {"prefix": {"dept": "Fin"}}, {"filtered": {"filter": {"range": {"join_date": {"gte": "2000-04-26T10:58:00", "lte": "2016-08-15T12:40:06.499612"}}}}}, {"filtered": {"filter": {"range": {"mutated": {"gte": 3, "lte": 2340}}}}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,111] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,111] - [task:1112] INFO - ------------------------------------------------------------------ Query # 57 ----------------------------------------------------------------- [2016-08-15 12:40:13,127] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Engineering"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,154] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,155] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Engineering"} is 0 (took 4.118702ms) [2016-08-15 12:40:13,159] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Engineering"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,159] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,159] - [task:1112] INFO - ------------------------------------------------------------------ Query # 58 ----------------------------------------------------------------- [2016-08-15 12:40:13,173] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 659, "inclusive_min": true, "min": 1}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,231] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,231] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 659, "inclusive_min": true, "min": 1} is 0 (took 35.481383ms) [2016-08-15 12:40:13,235] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"gte": 1, "lte": 659}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,235] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,235] - [task:1112] INFO - ------------------------------------------------------------------ Query # 59 ----------------------------------------------------------------- [2016-08-15 12:40:13,251] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.500763", "start": "2001-05-11T03:56:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,356] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,357] - [task:1140] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.500763", "start": "2001-05-11T03:56:00"} is 224 (took 68.743465ms) [2016-08-15 12:40:13,378] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"lt": "2016-08-15T12:40:06.500763", "gt": "2001-05-11T03:56:00"}}}}} on es_index is 224 (took 5ms) [2016-08-15 12:40:13,379] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,379] - [task:1112] INFO - ------------------------------------------------------------------ Query # 60 ----------------------------------------------------------------- [2016-08-15 12:40:13,397] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.501121", "start": "1953-01-12T11:51:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,527] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,528] - [task:1140] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.501121", "start": "1953-01-12T11:51:00"} is 939 (took 57.392698ms) [2016-08-15 12:40:13,595] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"gte": "1953-01-12T11:51:00", "lte": "2016-08-15T12:40:06.501121"}}}}} on es_index is 939 (took 16ms) [2016-08-15 12:40:13,596] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,597] - [task:1112] INFO - ------------------------------------------------------------------ Query # 61 ----------------------------------------------------------------- [2016-08-15 12:40:13,611] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Marketing"}, {"field": "name", "match": "Devi Jones"}, {"field": "name", "match": "Adena Ward"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,649] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,649] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Marketing"}, {"field": "name", "match": "Devi Jones"}, {"field": "name", "match": "Adena Ward"}]} is 5 (took 10.321546ms) [2016-08-15 12:40:13,654] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Marketing"}}, {"match": {"name": "Devi Jones"}}, {"match": {"name": "Adena Ward"}}]}} on es_index is 2 (took 1ms) [2016-08-15 12:40:13,654] - [task:1155] ERROR - FAIL: FTS hits: 5, while ES hits: 2 [2016-08-15 12:40:13,654] - [task:1170] ERROR - FAIL: Following 3 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000595', u'emp10000980', u'emp10000178'] [2016-08-15 12:40:13,655] - [task:1112] INFO - ------------------------------------------------------------------ Query # 62 ----------------------------------------------------------------- [2016-08-15 12:40:13,671] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "salary", "max": 126491.0, "inclusive_min": false, "min": 120000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,750] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,750] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "salary", "max": 126491.0, "inclusive_min": false, "min": 120000.0} is 56 (took 46.95453ms) [2016-08-15 12:40:13,758] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"lt": 126491.0, "gt": 120000.0}}}}} on es_index is 56 (took 2ms) [2016-08-15 12:40:13,759] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,759] - [task:1112] INFO - ------------------------------------------------------------------ Query # 63 ----------------------------------------------------------------- [2016-08-15 12:40:13,774] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "Ara"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,803] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,803] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "Ara"} is 29 (took 3.466633ms) [2016-08-15 12:40:13,808] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "Ara"}} on es_index is 15 (took 1ms) [2016-08-15 12:40:13,809] - [task:1155] ERROR - FAIL: FTS hits: 29, while ES hits: 15 [2016-08-15 12:40:13,809] - [task:1170] ERROR - FAIL: Following 14 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000051', u'emp10000704', u'emp10000764', u'emp10000076', u'emp10000591', u'emp10000507', u'emp10000745', u'emp10000010', u'emp10000769', u'emp10000369', u'emp10000830', u'emp10000590', u'emp10000915', u'emp10000474'] [2016-08-15 12:40:13,809] - [task:1112] INFO - ------------------------------------------------------------------ Query # 64 ----------------------------------------------------------------- [2016-08-15 12:40:13,829] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "-dept:\"Sales\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,931] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,932] - [task:1140] INFO - FTS hits for query: {"query": "-dept:\"Sales\""} is 1000 (took 60.117285ms) [2016-08-15 12:40:13,936] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "-dept:\"Sales\""}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,936] - [task:1155] ERROR - FAIL: FTS hits: 1000, while ES hits: 0 [2016-08-15 12:40:13,937] - [task:1170] ERROR - FAIL: Following 1000 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000538', u'emp10000539', u'emp10000536', u'emp10000537', u'emp10000534', u'emp10000535', u'emp10000532', u'emp10000533', u'emp10000530', u'emp10000531', u'emp10000437', u'emp10000124', u'emp10000435', u'emp10000434', u'emp10000121', u'emp10000120', u'emp10000431', u'emp10000122', u'emp10000719', u'emp10000129', u'emp10000128', u'emp10000439', u'emp10000438', u'emp10000168', u'emp10000472', u'emp10000441', u'emp10000466', u'emp10000467', u'emp10000471', u'emp10000721', u'emp10000417', u'emp10000222', u'emp10000223', u'emp10000549', u'emp10000548', u'emp10000226', u'emp10000227', u'emp10000224', u'emp10000225', u'emp10000543', u'emp10000542', u'emp10000228', u'emp10000229', u'emp10000547', u'emp10000546', u'emp10000545', u'emp10000544', u'emp10000983', u'emp10000982', u'emp10000981'] [2016-08-15 12:40:13,937] - [task:1112] INFO - ------------------------------------------------------------------ Query # 65 ----------------------------------------------------------------- [2016-08-15 12:40:13,956] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match_phrase": "Solita Mitchell"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:13,987] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:13,987] - [task:1140] INFO - FTS hits for query: {"field": "name", "match_phrase": "Solita Mitchell"} is 0 (took 4.073042ms) [2016-08-15 12:40:13,993] - [task:1150] INFO - ES hits for query: {"match_phrase": {"name": "Solita Mitchell"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:13,994] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:13,994] - [task:1112] INFO - ------------------------------------------------------------------ Query # 66 ----------------------------------------------------------------- [2016-08-15 12:40:14,006] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Accounts"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,031] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,032] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Accounts"} is 0 (took 4.493109ms) [2016-08-15 12:40:14,035] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Accounts"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,035] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,035] - [task:1112] INFO - ------------------------------------------------------------------ Query # 67 ----------------------------------------------------------------- [2016-08-15 12:40:14,049] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "wildcard": "Su*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,078] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,078] - [task:1140] INFO - FTS hits for query: {"field": "dept", "wildcard": "Su*"} is 84 (took 5.011399ms) [2016-08-15 12:40:14,082] - [task:1150] INFO - ES hits for query: {"wildcard": {"dept": "Su*"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,082] - [task:1155] ERROR - FAIL: FTS hits: 84, while ES hits: 0 [2016-08-15 12:40:14,083] - [task:1170] ERROR - FAIL: Following 84 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000292', u'emp10000163', u'emp10000650', u'emp10000297', u'emp10000670', u'emp10000673', u'emp10000652', u'emp10000867', u'emp10000273', u'emp10000412', u'emp10000275', u'emp10000007', u'emp10000063', u'emp10000102', u'emp10000479', u'emp10000302', u'emp10000341', u'emp10000106', u'emp10000105', u'emp10000234', u'emp10000369', u'emp10000904', u'emp10000217', u'emp10000798', u'emp10000626', u'emp10000646', u'emp10000978', u'emp10000955', u'emp10000976', u'emp10000886', u'emp10000882', u'emp10000724', u'emp10000346', u'emp10000877', u'emp10000627', u'emp10000742', u'emp10000819', u'emp10000545', u'emp10000481', u'emp10000911', u'emp10000913', u'emp10000582', u'emp10000602', u'emp10000629', u'emp10000281', u'emp10000315', u'emp10000173', u'emp10000703', u'emp10000171', u'emp10000418'] [2016-08-15 12:40:14,083] - [task:1112] INFO - ------------------------------------------------------------------ Query # 68 ----------------------------------------------------------------- [2016-08-15 12:40:14,097] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "B"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,131] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,132] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "B"} is 153 (took 9.902373ms) [2016-08-15 12:40:14,141] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "B"}} on es_index is 70 (took 3ms) [2016-08-15 12:40:14,142] - [task:1155] ERROR - FAIL: FTS hits: 153, while ES hits: 70 [2016-08-15 12:40:14,142] - [task:1170] ERROR - FAIL: Following 84 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000293', u'emp10000732', u'emp10000089', u'emp10000670', u'emp10000327', u'emp10000326', u'emp10000142', u'emp10000058', u'emp10000792', u'emp10000455', u'emp10000022', u'emp10000254', u'emp10000123', u'emp10000575', u'emp10000028', u'emp10000380', u'emp10000369', u'emp10000382', u'emp10000385', u'emp10000230', u'emp10000233', u'emp10000285', u'emp10000990', u'emp10000579', u'emp10000740', u'emp10000998', u'emp10000886', u'emp10000852', u'emp10000854', u'emp10000970', u'emp10000077', u'emp10000624', u'emp10000710', u'emp10000439', u'emp10000952', u'emp10000621', u'emp10000910', u'emp10000263', u'emp10000914', u'emp10000582', u'emp10000879', u'emp10000878', u'emp10000768', u'emp10000199', u'emp10000173', u'emp10000268', u'emp10000171', u'emp10000648', u'emp10000133', u'emp10000930'] [2016-08-15 12:40:14,143] - [task:1112] INFO - ------------------------------------------------------------------ Query # 69 ----------------------------------------------------------------- [2016-08-15 12:40:14,154] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Training"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,178] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,178] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Training"} is 0 (took 3.842067ms) [2016-08-15 12:40:14,182] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Training"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,182] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,182] - [task:1112] INFO - ------------------------------------------------------------------ Query # 70 ----------------------------------------------------------------- [2016-08-15 12:40:14,199] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "-dept:\"Support\" +dept:\"Pre-sales\" -name:\"Sirena Wood\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,231] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,232] - [task:1140] INFO - FTS hits for query: {"query": "-dept:\"Support\" +dept:\"Pre-sales\" -name:\"Sirena Wood\""} is 0 (took 9.006397ms) [2016-08-15 12:40:14,302] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "-dept:\"Support\" +dept:\"Pre-sales\" -name:\"Sirena Wood\""}} on es_index is 1000 (took 16ms) [2016-08-15 12:40:14,302] - [task:1155] ERROR - FAIL: FTS hits: 0, while ES hits: 1000 [2016-08-15 12:40:14,303] - [task:1170] ERROR - FAIL: Following 1000 docs were not returned by FTS, but ES, printing 50: [u'emp10000538', u'emp10000539', u'emp10000536', u'emp10000537', u'emp10000534', u'emp10000535', u'emp10000532', u'emp10000533', u'emp10000530', u'emp10000531', u'emp10000125', u'emp10000436', u'emp10000435', u'emp10000126', u'emp10000121', u'emp10000120', u'emp10000431', u'emp10000430', u'emp10000129', u'emp10000128', u'emp10000439', u'emp10000438', u'emp10000509', u'emp10000508', u'emp10000192', u'emp10000471', u'emp10000721', u'emp10000417', u'emp10000222', u'emp10000223', u'emp10000549', u'emp10000221', u'emp10000226', u'emp10000227', u'emp10000224', u'emp10000225', u'emp10000543', u'emp10000542', u'emp10000228', u'emp10000229', u'emp10000547', u'emp10000546', u'emp10000545', u'emp10000544', u'emp10000983', u'emp10000982', u'emp10000981', u'emp10000980', u'emp10000987', u'emp10000986'] [2016-08-15 12:40:14,304] - [task:1112] INFO - ------------------------------------------------------------------ Query # 71 ----------------------------------------------------------------- [2016-08-15 12:40:14,320] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"should": {"disjuncts": [{"field": "dept", "match": "Accounts"}, {"field": "name", "match": "Adrianne Juan José"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,354] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,354] - [task:1140] INFO - FTS hits for query: {"should": {"disjuncts": [{"field": "dept", "match": "Accounts"}, {"field": "name", "match": "Adrianne Juan José"}]}} is 14 (took 8.348853ms) [2016-08-15 12:40:14,361] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Accounts"}}, {"match": {"name": "Adrianne Juan José"}}]}} on es_index is 12 (took 3ms) [2016-08-15 12:40:14,362] - [task:1155] ERROR - FAIL: FTS hits: 14, while ES hits: 12 [2016-08-15 12:40:14,362] - [task:1170] ERROR - FAIL: Following 2 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000933', u'emp10000871'] [2016-08-15 12:40:14,362] - [task:1112] INFO - ------------------------------------------------------------------ Query # 72 ----------------------------------------------------------------- [2016-08-15 12:40:14,378] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match": "Araceli Reed"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,402] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,402] - [task:1140] INFO - FTS hits for query: {"field": "name", "match": "Araceli Reed"} is 0 (took 4.564501ms) [2016-08-15 12:40:14,406] - [task:1150] INFO - ES hits for query: {"match": {"name": "Araceli Reed"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,407] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,407] - [task:1112] INFO - ------------------------------------------------------------------ Query # 73 ----------------------------------------------------------------- [2016-08-15 12:40:14,424] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "prefix": "Su"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,461] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,462] - [task:1140] INFO - FTS hits for query: {"field": "dept", "prefix": "Su"} is 84 (took 10.002174ms) [2016-08-15 12:40:14,466] - [task:1150] INFO - ES hits for query: {"prefix": {"dept": "Su"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,466] - [task:1155] ERROR - FAIL: FTS hits: 84, while ES hits: 0 [2016-08-15 12:40:14,466] - [task:1170] ERROR - FAIL: Following 84 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000292', u'emp10000163', u'emp10000650', u'emp10000297', u'emp10000670', u'emp10000673', u'emp10000652', u'emp10000867', u'emp10000273', u'emp10000412', u'emp10000275', u'emp10000007', u'emp10000063', u'emp10000102', u'emp10000479', u'emp10000302', u'emp10000341', u'emp10000106', u'emp10000105', u'emp10000234', u'emp10000369', u'emp10000904', u'emp10000217', u'emp10000798', u'emp10000626', u'emp10000703', u'emp10000978', u'emp10000955', u'emp10000976', u'emp10000886', u'emp10000882', u'emp10000724', u'emp10000346', u'emp10000877', u'emp10000627', u'emp10000742', u'emp10000819', u'emp10000545', u'emp10000481', u'emp10000911', u'emp10000913', u'emp10000582', u'emp10000602', u'emp10000629', u'emp10000281', u'emp10000315', u'emp10000173', u'emp10000646', u'emp10000171', u'emp10000418'] [2016-08-15 12:40:14,467] - [task:1112] INFO - ------------------------------------------------------------------ Query # 74 ----------------------------------------------------------------- [2016-08-15 12:40:14,479] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Caryssa Bail*"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,505] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,505] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Caryssa Bail*"} is 0 (took 2.068983ms) [2016-08-15 12:40:14,509] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Caryssa Bail*"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,509] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,509] - [task:1112] INFO - ------------------------------------------------------------------ Query # 75 ----------------------------------------------------------------- [2016-08-15 12:40:14,523] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match_phrase": "Killian Tomás"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,571] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,571] - [task:1140] INFO - FTS hits for query: {"field": "name", "match_phrase": "Killian Tomás"} is 0 (took 27.617927ms) [2016-08-15 12:40:14,575] - [task:1150] INFO - ES hits for query: {"match_phrase": {"name": "Killian Tomás"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,575] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,575] - [task:1112] INFO - ------------------------------------------------------------------ Query # 76 ----------------------------------------------------------------- [2016-08-15 12:40:14,590] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.505749", "start": "1980-06-20T06:01:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,715] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,716] - [task:1140] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.505749", "start": "1980-06-20T06:01:00"} is 522 (took 70.623594ms) [2016-08-15 12:40:14,757] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"lt": "2016-08-15T12:40:06.505749", "gt": "1980-06-20T06:01:00"}}}}} on es_index is 522 (took 10ms) [2016-08-15 12:40:14,758] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:14,759] - [task:1112] INFO - ------------------------------------------------------------------ Query # 77 ----------------------------------------------------------------- [2016-08-15 12:40:14,774] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "Wel"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,804] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,805] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "Wel"} is 16 (took 2.708223ms) [2016-08-15 12:40:14,809] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "Wel"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:14,809] - [task:1155] ERROR - FAIL: FTS hits: 16, while ES hits: 0 [2016-08-15 12:40:14,809] - [task:1170] ERROR - FAIL: Following 16 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000730', u'emp10000630', u'emp10000976', u'emp10000270', u'emp10000377', u'emp10000091', u'emp10000388', u'emp10000092', u'emp10000802', u'emp10000573', u'emp10000101', u'emp10000867', u'emp10000568', u'emp10000308', u'emp10000102', u'emp10000477'] [2016-08-15 12:40:14,810] - [task:1112] INFO - ------------------------------------------------------------------ Query # 78 ----------------------------------------------------------------- [2016-08-15 12:40:14,825] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "prefix": "M"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,876] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,877] - [task:1140] INFO - FTS hits for query: {"field": "name", "prefix": "M"} is 252 (took 18.350094ms) [2016-08-15 12:40:14,885] - [task:1150] INFO - ES hits for query: {"prefix": {"name": "M"}} on es_index is 49 (took 2ms) [2016-08-15 12:40:14,885] - [task:1155] ERROR - FAIL: FTS hits: 252, while ES hits: 49 [2016-08-15 12:40:14,885] - [task:1170] ERROR - FAIL: Following 204 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000675', u'emp10000042', u'emp10000329', u'emp10000041', u'emp10000187', u'emp10000598', u'emp10000320', u'emp10000437', u'emp10000431', u'emp10000734', u'emp10000230', u'emp10000540', u'emp10000627', u'emp10000547', u'emp10000974', u'emp10000973', u'emp10000971', u'emp10000875', u'emp10000870', u'emp10000878', u'emp10000704', u'emp10000707', u'emp10000701', u'emp10000339', u'emp10000747', u'emp10000336', u'emp10000708', u'emp10000333', u'emp10000955', u'emp10000037', u'emp10000788', u'emp10000481', u'emp10000787', u'emp10000786', u'emp10000781', u'emp10000780', u'emp10000980', u'emp10000984', u'emp10000865', u'emp10000908', u'emp10000637', u'emp10000419', u'emp10000300', u'emp10000658', u'emp10000415', u'emp10000417', u'emp10000140', u'emp10000558', u'emp10000021', u'emp10000025'] [2016-08-15 12:40:14,886] - [task:1112] INFO - ------------------------------------------------------------------ Query # 79 ----------------------------------------------------------------- [2016-08-15 12:40:14,900] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "dept", "match": "Marketing"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.506703", "start": "1974-08-25T14:23:00"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:14,999] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:14,999] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "dept", "match": "Marketing"}, {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.506703", "start": "1974-08-25T14:23:00"}]} is 0 (took 71.542695ms) [2016-08-15 12:40:15,003] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"dept": "Marketing"}}, {"filtered": {"filter": {"range": {"join_date": {"gte": "1974-08-25T14:23:00", "lte": "2016-08-15T12:40:06.506703"}}}}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,003] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,004] - [task:1112] INFO - ------------------------------------------------------------------ Query # 80 ----------------------------------------------------------------- [2016-08-15 12:40:15,023] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 101265.0, "inclusive_min": true, "min": 92000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,085] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,086] - [task:1140] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 101265.0, "inclusive_min": true, "min": 92000.0} is 93 (took 31.562893ms) [2016-08-15 12:40:15,096] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"gte": 92000.0, "lte": 101265.0}}}}} on es_index is 93 (took 2ms) [2016-08-15 12:40:15,097] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,097] - [task:1112] INFO - ------------------------------------------------------------------ Query # 81 ----------------------------------------------------------------- [2016-08-15 12:40:15,111] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"should": {"disjuncts": [{"field": "dept", "match": "Dev-ops"}]}, "must": {"conjuncts": [{"field": "name", "match": "Keelia Roberts"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,135] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,136] - [task:1140] INFO - FTS hits for query: {"should": {"disjuncts": [{"field": "dept", "match": "Dev-ops"}]}, "must": {"conjuncts": [{"field": "name", "match": "Keelia Roberts"}]}} is 0 (took 4.736112ms) [2016-08-15 12:40:15,140] - [task:1150] INFO - ES hits for query: {"bool": {"should": {"match": {"dept": "Dev-ops"}}, "must": {"match": {"name": "Keelia Roberts"}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,140] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,140] - [task:1112] INFO - ------------------------------------------------------------------ Query # 82 ----------------------------------------------------------------- [2016-08-15 12:40:15,154] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"should": {"disjuncts": [{"field": "dept", "match": "Accounts"}, {"field": "name", "match": "Martin"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,181] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,181] - [task:1140] INFO - FTS hits for query: {"should": {"disjuncts": [{"field": "dept", "match": "Accounts"}, {"field": "name", "match": "Martin"}]}} is 3 (took 8.28645ms) [2016-08-15 12:40:15,185] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Accounts"}}, {"match": {"name": "Martin"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,185] - [task:1155] ERROR - FAIL: FTS hits: 3, while ES hits: 0 [2016-08-15 12:40:15,185] - [task:1170] ERROR - FAIL: Following 3 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000874', u'emp10000137', u'emp10000480'] [2016-08-15 12:40:15,186] - [task:1112] INFO - ------------------------------------------------------------------ Query # 83 ----------------------------------------------------------------- [2016-08-15 12:40:15,201] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "mutated", "max": 1860, "inclusive_min": false, "min": 5}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,275] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,275] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "mutated", "max": 1860, "inclusive_min": false, "min": 5} is 0 (took 43.175156ms) [2016-08-15 12:40:15,280] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"lt": 1860, "gt": 5}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,280] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,280] - [task:1112] INFO - ------------------------------------------------------------------ Query # 84 ----------------------------------------------------------------- [2016-08-15 12:40:15,295] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"must_not": {"disjuncts": [{"field": "dept", "match": "Marketing"}]}, "should": {"disjuncts": [{"field": "name", "match": "Dabria Simón"}, {"field": "name", "match": "Adara Anderson"}, {"field": "dept", "match": "HR"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,330] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,331] - [task:1140] INFO - FTS hits for query: {"must_not": {"disjuncts": [{"field": "dept", "match": "Marketing"}]}, "should": {"disjuncts": [{"field": "name", "match": "Dabria Simón"}, {"field": "name", "match": "Adara Anderson"}, {"field": "dept", "match": "HR"}]}} is 1 (took 9.377306ms) [2016-08-15 12:40:15,336] - [task:1150] INFO - ES hits for query: {"bool": {"must_not": {"match": {"dept": "Marketing"}}, "should": [{"match": {"name": "Dabria Simón"}}, {"match": {"name": "Adara Anderson"}}, {"match": {"dept": "HR"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,337] - [task:1155] ERROR - FAIL: FTS hits: 1, while ES hits: 0 [2016-08-15 12:40:15,337] - [task:1170] ERROR - FAIL: Following 1 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000230'] [2016-08-15 12:40:15,337] - [task:1112] INFO - ------------------------------------------------------------------ Query # 85 ----------------------------------------------------------------- [2016-08-15 12:40:15,353] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Sco??"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,382] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,382] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Sco??"} is 0 (took 3.53382ms) [2016-08-15 12:40:15,388] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Sco??"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,388] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,389] - [task:1112] INFO - ------------------------------------------------------------------ Query # 86 ----------------------------------------------------------------- [2016-08-15 12:40:15,410] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match_phrase": "Mercedes Baker III"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,436] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,437] - [task:1140] INFO - FTS hits for query: {"field": "name", "match_phrase": "Mercedes Baker III"} is 0 (took 3.411861ms) [2016-08-15 12:40:15,440] - [task:1150] INFO - ES hits for query: {"match_phrase": {"name": "Mercedes Baker III"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,441] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,441] - [task:1112] INFO - ------------------------------------------------------------------ Query # 87 ----------------------------------------------------------------- [2016-08-15 12:40:15,454] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match": "Engineering"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,481] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,481] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match": "Engineering"} is 0 (took 3.967545ms) [2016-08-15 12:40:15,484] - [task:1150] INFO - ES hits for query: {"match": {"dept": "Engineering"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,484] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,485] - [task:1112] INFO - ------------------------------------------------------------------ Query # 88 ----------------------------------------------------------------- [2016-08-15 12:40:15,501] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.509702", "start": "1988-05-03T06:17:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,592] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,593] - [task:1140] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2016-08-15T12:40:06.509702", "start": "1988-05-03T06:17:00"} is 421 (took 49.877729ms) [2016-08-15 12:40:15,626] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"gte": "1988-05-03T06:17:00", "lte": "2016-08-15T12:40:06.509702"}}}}} on es_index is 421 (took 8ms) [2016-08-15 12:40:15,627] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,628] - [task:1112] INFO - ------------------------------------------------------------------ Query # 89 ----------------------------------------------------------------- [2016-08-15 12:40:15,643] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "+mutated:>=0"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,766] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,767] - [task:1140] INFO - FTS hits for query: {"query": "+mutated:>=0"} is 1000 (took 41.548601ms) [2016-08-15 12:40:15,851] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "+mutated:>=0"}} on es_index is 1000 (took 17ms) [2016-08-15 12:40:15,852] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,853] - [task:1112] INFO - ------------------------------------------------------------------ Query # 90 ----------------------------------------------------------------- [2016-08-15 12:40:15,871] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "wildcard": "Ro?ers"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:15,900] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:15,900] - [task:1140] INFO - FTS hits for query: {"field": "name", "wildcard": "Ro?ers"} is 0 (took 6.255246ms) [2016-08-15 12:40:15,905] - [task:1150] INFO - ES hits for query: {"wildcard": {"name": "Ro?ers"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:15,905] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:15,905] - [task:1112] INFO - ------------------------------------------------------------------ Query # 91 ----------------------------------------------------------------- [2016-08-15 12:40:15,922] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.510734", "start": "1988-08-15T09:38:00"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,043] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,044] - [task:1140] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2016-08-15T12:40:06.510734", "start": "1988-08-15T09:38:00"} is 417 (took 74.853964ms) [2016-08-15 12:40:16,107] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"join_date": {"lt": "2016-08-15T12:40:06.510734", "gt": "1988-08-15T09:38:00"}}}}} on es_index is 417 (took 12ms) [2016-08-15 12:40:16,108] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,109] - [task:1112] INFO - ------------------------------------------------------------------ Query # 92 ----------------------------------------------------------------- [2016-08-15 12:40:16,129] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "match_phrase": "Marketing"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,160] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,160] - [task:1140] INFO - FTS hits for query: {"field": "dept", "match_phrase": "Marketing"} is 0 (took 3.218175ms) [2016-08-15 12:40:16,169] - [task:1150] INFO - ES hits for query: {"match_phrase": {"dept": "Marketing"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,170] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,170] - [task:1112] INFO - ------------------------------------------------------------------ Query # 93 ----------------------------------------------------------------- [2016-08-15 12:40:16,192] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"should": {"disjuncts": [{"field": "name", "match": "Lysha Jones"}, {"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Jr."}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Kory Lewis"}]}, "must": {"conjuncts": [{"field": "name", "match": "Devi Carter"}, {"field": "name", "match": "Kagami"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,230] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,231] - [task:1140] INFO - FTS hits for query: {"should": {"disjuncts": [{"field": "name", "match": "Lysha Jones"}, {"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Jr."}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Kory Lewis"}]}, "must": {"conjuncts": [{"field": "name", "match": "Devi Carter"}, {"field": "name", "match": "Kagami"}]}} is 0 (took 10.538558ms) [2016-08-15 12:40:16,236] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "Lysha Jones"}}, {"match": {"dept": "Engineering"}}, {"match": {"name": "Jr."}}], "must_not": {"match": {"name": "Kory Lewis"}}, "must": [{"match": {"name": "Devi Carter"}}, {"match": {"name": "Kagami"}}]}} on es_index is 0 (took 2ms) [2016-08-15 12:40:16,237] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,237] - [task:1112] INFO - ------------------------------------------------------------------ Query # 94 ----------------------------------------------------------------- [2016-08-15 12:40:16,251] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "dept", "prefix": "E"}, {"field": "name", "match_phrase": "Cynara Howard Jr."}, {"inclusive_max": false, "field": "mutated", "max": 5729, "inclusive_min": false, "min": 0}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,342] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,342] - [task:1140] INFO - FTS hits for query: {"conjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "dept", "prefix": "E"}, {"field": "name", "match_phrase": "Cynara Howard Jr."}, {"inclusive_max": false, "field": "mutated", "max": 5729, "inclusive_min": false, "min": 0}]} is 0 (took 63.477425ms) [2016-08-15 12:40:16,347] - [task:1150] INFO - ES hits for query: {"bool": {"must": [{"match": {"dept": "Engineering"}}, {"prefix": {"dept": "E"}}, {"match_phrase": {"name": "Cynara Howard Jr."}}, {"filtered": {"filter": {"range": {"mutated": {"lt": 5729, "gt": 0}}}}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,347] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,347] - [task:1112] INFO - ------------------------------------------------------------------ Query # 95 ----------------------------------------------------------------- [2016-08-15 12:40:16,362] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "salary", "max": 84684.0, "inclusive_min": false, "min": 83000.0}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,418] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,419] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "salary", "max": 84684.0, "inclusive_min": false, "min": 83000.0} is 19 (took 30.488684ms) [2016-08-15 12:40:16,424] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"salary": {"lt": 84684.0, "gt": 83000.0}}}}} on es_index is 19 (took 1ms) [2016-08-15 12:40:16,425] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,425] - [task:1112] INFO - ------------------------------------------------------------------ Query # 96 ----------------------------------------------------------------- [2016-08-15 12:40:16,439] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"query": "dept:\"Info-tech\""}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,461] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,462] - [task:1140] INFO - FTS hits for query: {"query": "dept:\"Info-tech\""} is 0 (took 3.615304ms) [2016-08-15 12:40:16,466] - [task:1150] INFO - ES hits for query: {"query_string": {"query": "dept:\"Info-tech\""}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,466] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,467] - [task:1112] INFO - ------------------------------------------------------------------ Query # 97 ----------------------------------------------------------------- [2016-08-15 12:40:16,481] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": false, "field": "mutated", "max": 7896, "inclusive_min": false, "min": 2}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,561] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,562] - [task:1140] INFO - FTS hits for query: {"inclusive_max": false, "field": "mutated", "max": 7896, "inclusive_min": false, "min": 2} is 0 (took 58.010013ms) [2016-08-15 12:40:16,567] - [task:1150] INFO - ES hits for query: {"filtered": {"filter": {"range": {"mutated": {"lt": 7896, "gt": 2}}}}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,567] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,567] - [task:1112] INFO - ------------------------------------------------------------------ Query # 98 ----------------------------------------------------------------- [2016-08-15 12:40:16,581] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"should": {"disjuncts": [{"field": "name", "match": "Drucilla"}, {"field": "dept", "match": "Dev-ops"}]}}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,618] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,618] - [task:1140] INFO - FTS hits for query: {"should": {"disjuncts": [{"field": "name", "match": "Drucilla"}, {"field": "dept", "match": "Dev-ops"}]}} is 0 (took 5.326504ms) [2016-08-15 12:40:16,622] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"name": "Drucilla"}}, {"match": {"dept": "Dev-ops"}}]}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,622] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified [2016-08-15 12:40:16,623] - [task:1112] INFO - ------------------------------------------------------------------ Query # 99 ----------------------------------------------------------------- [2016-08-15 12:40:16,646] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Marketing"}, {"field": "dept", "match": "Pre-sales"}, {"field": "name", "match": "Kallie Russell"}]}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,675] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,676] - [task:1140] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Marketing"}, {"field": "dept", "match": "Pre-sales"}, {"field": "name", "match": "Kallie Russell"}]} is 6 (took 6.958317ms) [2016-08-15 12:40:16,683] - [task:1150] INFO - ES hits for query: {"bool": {"should": [{"match": {"dept": "Marketing"}}, {"match": {"dept": "Pre-sales"}}, {"match": {"name": "Kallie Russell"}}]}} on es_index is 1 (took 3ms) [2016-08-15 12:40:16,684] - [task:1155] ERROR - FAIL: FTS hits: 6, while ES hits: 1 [2016-08-15 12:40:16,684] - [task:1170] ERROR - FAIL: Following 5 doc(s) were not returned by ES,but FTS, printing 50: [u'emp10000932', u'emp10000487', u'emp10000502', u'emp10000772', u'emp10000870'] [2016-08-15 12:40:16,684] - [task:1112] INFO - ------------------------------------------------------------------ Query # 100 ----------------------------------------------------------------- [2016-08-15 12:40:16,707] - [fts_base:1173] INFO - Running query {"from": 0, "indexName": "custom_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "dept", "wildcard": "De?-ops"}, "size": 10000000} on node: 172.23.106.72: [2016-08-15 12:40:16,735] - [task:1116] INFO - Status: {u'successful': 32, u'failed': 0, u'total': 32} [2016-08-15 12:40:16,737] - [task:1140] INFO - FTS hits for query: {"field": "dept", "wildcard": "De?-ops"} is 0 (took 3.865529ms) [2016-08-15 12:40:16,741] - [task:1150] INFO - ES hits for query: {"wildcard": {"dept": "De?-ops"}} on es_index is 0 (took 1ms) [2016-08-15 12:40:16,742] - [task:1159] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified FAIL [2016-08-15 12:40:16,766] - [rest_client:1299] INFO - /diag/eval status on 172.23.106.72:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-08-15 12:40:16,769] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:40:17,054] - [remote_util:206] INFO - Connected to 172.23.106.72 [2016-08-15 12:40:17,719] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-08-15 12:40:18,555] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:40:18,556] - [fts_base:2474] INFO - 0 [2016-08-15 12:40:18,556] - [fts_base:2482] INFO - Initial 'panic:' count on 172.23.106.72 :0, now :0 [2016-08-15 12:40:18,556] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-08-15 12:40:19,424] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:40:19,425] - [fts_base:2474] INFO - 0 [2016-08-15 12:40:19,425] - [fts_base:2482] INFO - Initial 'crash in forestdb' count on 172.23.106.72 :0, now :0 [2016-08-15 12:40:19,540] - [fts_base:2047] INFO - Collecting logs @ 172.23.106.72 grabbing cbcollect from 172.23.106.72 [2016-08-15 12:40:19,542] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:40:19,828] - [remote_util:206] INFO - Connected to 172.23.106.72 Collecting logs from 172.23.106.72 [2016-08-15 12:40:20,530] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: /opt/couchbase/bin/cbcollect_info 172.23.106.72-20160815-1240-diag.zip [2016-08-15 12:43:09,136] - [remote_util:2420] INFO - command executed successfully Downloading zipped logs from 172.23.106.72 [2016-08-15 12:43:10,212] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: rm -f /root/172.23.106.72-20160815-1240-diag.zip [2016-08-15 12:43:10,275] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:43:10,278] - [fts_base:2056] INFO - ==== FTSbasetests cleanup is started for test #60 index_query_custom_mapping ==== [2016-08-15 12:43:10,312] - [fts_base:663] INFO - Deleting fulltext-index custom_index on 172.23.106.72 [2016-08-15 12:43:10,361] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:43:10,640] - [remote_util:206] INFO - Connected to 172.23.106.72 [2016-08-15 12:43:11,315] - [remote_util:2383] INFO - running command.raw on 172.23.106.72: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep custom_index*.pindex | wc -l [2016-08-15 12:43:11,422] - [remote_util:2420] INFO - command executed successfully [2016-08-15 12:43:11,423] - [fts_base:886] INFO - 0 [2016-08-15 12:43:11,537] - [fts_base:670] INFO - Validated: all index files for custom_index deleted from disk [2016-08-15 12:43:11,538] - [fts_base:978] INFO - removing nodes from cluster ... [2016-08-15 12:43:11,552] - [fts_base:980] INFO - cleanup [ip:172.23.106.72 port:8091 ssh_username:root] [2016-08-15 12:43:11,585] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.106.72 [2016-08-15 12:43:11,585] - [bucket_helper:140] INFO - remove bucket default ... [2016-08-15 12:43:14,016] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.106.72 [2016-08-15 12:43:14,016] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-08-15 12:43:14,026] - [rest_client:133] INFO - node 172.23.106.72 existing buckets : [] [2016-08-15 12:43:14,081] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.106.72:8091 [2016-08-15 12:43:14,096] - [cluster_helper:80] INFO - ns_server @ 172.23.106.72:8091 is running Cluster instance shutdown with force [2016-08-15 12:43:14,122] - [es_base:208] INFO - ES index es_index deleted [2016-08-15 12:43:14,122] - [fts_base:2062] INFO - ==== FTSbasetests cleanup is finished for test #60 index_query_custom_mapping === ====================================================================== FAIL: index_query_custom_mapping (fts.stable_topology_fts.StableTopFTS) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/fts/stable_topology_fts.py", line 339, in index_query_custom_mapping self.run_query_and_compare(index) File "pytests/fts/fts_base.py", line 2941, in run_query_and_compare failed_queries)) AssertionError: 32 out of 100 queries failed! - [4, 7, 8, 9, 10, 11, 13, 16, 20, 23, 28, 29, 35, 38, 40, 44, 47, 49, 55, 61, 63, 64, 67, 68, 70, 71, 73, 77, 78, 82, 84, 99] ---------------------------------------------------------------------- Ran 1 test in 217.023s FAILED (failures=1) Cluster instance shutdown with force grabbing core dumps files from 172.23.106.72 [2016-08-15 12:43:14,124] - [remote_util:172] INFO - connecting to 172.23.106.72 with username : root password : couchbase ssh_key: [2016-08-15 12:43:14,419] - [remote_util:206] INFO - Connected to 172.23.106.72 [2016-08-15 12:43:15,239] - [remote_util:481] INFO - 172.23.106.72 **** The version file /opt/couchbase/ VERSION.txt exists downloading 172.23.106.72 .........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................looking for crashes on 172.23.106.72 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ breakpad *dmp files under /opt/couchbase/var/lib/couchbase/ crashes not found on 172.23.106.72 grabbing diags from http://172.23.106.72:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-p0-fts-vset00-00-custom-map-rqg/logs/testrunner-16-Aug-15_11-29-36/test_60/172.23.106.72-diag.txt.gz summary so far suite fts.stable_topology_fts.StableTopFTS , pass 52 , fail 8 failures so far... fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping testrunner logs, diags and results are available under /data/workspace/cen006-p0-fts-vset00-00-custom-map-rqg/logs/testrunner-16-Aug-15_11-29-36/test_60