filename: conf/fts/test.conf prefix: fts.stable_topology_fts.StableTopFTS Global Test input params: {'cluster_name': 'fts', 'conf_file': 'conf/fts/test.conf', 'ini': 'fts.ini', 'num_nodes': 2, 'spec': 'test'} Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_1 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=salary,expected=emp10000042,emp10000001,emp10000071,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'salary', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 1, 'expected': 'emp10000042,emp10000001,emp10000071', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_1', 'spec': 'test'} Run before suite setup for fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields 2016-11-04 12:33:27 | WARNING | MainProcess | MainThread | [data_helper.] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [fts_base.setUp] ==== FTSbasetests setup is started for test #1 test_sorting_of_results_on_non_indexed_fields ==== 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] cleanup [ip:10.111.151.101 port:8091 ssh_username:root] 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.151.101 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.151.101:8091 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.151.101:8091 is running 2016-11-04 12:33:27 | INFO | MainProcess | test_thread | [fts_base.init_cluster] Initializing Cluster ... 2016-11-04 12:33:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1539 2016-11-04 12:33:28 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb 2016-11-04 12:33:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 2016-11-04 12:33:28 | INFO | MainProcess | test_thread | [fts_base.init_cluster] 10.111.151.102 will be configured with services fts 2016-11-04 12:33:29 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.111.151.102:8091 to cluster 2016-11-04 12:33:29 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 2016-11-04 12:33:32 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 2016-11-04 12:33:32 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2016-11-04 12:33:32 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2016-11-04 12:33:42 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0486290455 sec 2016-11-04 12:33:43 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None 2016-11-04 12:33:43 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2016-11-04 12:33:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.111.151.101 to accept set ops 2016-11-04 12:33:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.151.101:11210 default 2016-11-04 12:33:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.151.101:11210 default 2016-11-04 12:33:46 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1039 2016-11-04 12:33:46 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2016-11-04 12:33:46 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.101 with username : root password : couchbase ssh_key: 2016-11-04 12:33:46 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.101 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.102 with username : root password : couchbase ssh_key: 2016-11-04 12:33:47 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.102 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.__initialize_error_count_dict] {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.setUp] ==== FTSbasetests setup is finished for test #1 test_sorting_of_results_on_non_indexed_fields ==== 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.111.151.101:11210 default 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.load_data] Loading phase complete! 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.create] Checking if index already exists ... 2016-11-04 12:33:48 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found 2016-11-04 12:33:48 | ERROR | MainProcess | test_thread | [rest_client._http_request] http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index default_index on 10.111.151.101 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] Index default_index created 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validating index distribution for default_index ... 2016-11-04 12:33:48 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 5 secs. No pindexes found, waiting for index to get created ... 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 32 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 32 partitions or lesser 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: pIndexes are distributed across [u'011ccecb1432b991dcabff4a0da485c9', u'2630a720f5f7aa02c8b227348ee0ae13'] 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 480-544 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node 2630a720f5f7aa02c8b227348ee0ae13 houses 16 pindexes which serve 512 partitions 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100, docs in FTS index 'default_index': 100 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] FTS indexed 100 docs in 0.0 mins 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.update] Updating fulltext-index default_index on 10.111.151.102 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [rest_client.update_fts_index] { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "3bffe65dfdddf637" } 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [rest_client.update_fts_index] Index/alias default_index updated 2016-11-04 12:33:54 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 5 secs. ... 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100, docs in FTS index 'default_index': 100 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] FTS indexed 100 docs in 0.0 mins 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.run_fts_query] Running query {"sort": ["salary"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [stable_topology_fts.test_sorting_of_results_on_non_indexed_fields] Hits: 3 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [stable_topology_fts.test_sorting_of_results_on_non_indexed_fields] Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_6d05b0c7b417ba95_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_6d05b0c7b417ba95_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_6d05b0c7b417ba95_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}] 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.validate_sorted_results] Actual docs returned : [u'emp10000001', u'emp10000071', u'emp10000042'] 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.validate_sorted_results] Expected docs : ['emp10000042', 'emp10000001', 'emp10000071'] 2016-11-04 12:33:59 | ERROR | MainProcess | test_thread | [stable_topology_fts.test_sorting_of_results_on_non_indexed_fields] Testcase failed. Actual results do not match expected. 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.101 with username : root password : couchbase ssh_key: 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.101 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] Initial 'panic:' count on 10.111.151.101 :0, now :0 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.102 with username : root password : couchbase ssh_key: 2016-11-04 12:33:59 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.102 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] Initial 'panic:' count on 10.111.151.102 :0, now :0 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] 0 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.check_error_count_in_fts_log] Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.tearDown] ==== FTSbasetests cleanup is started for test #1 test_sorting_of_results_on_non_indexed_fields ==== 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [fts_base.delete] Deleting fulltext-index default_index on 10.111.151.102 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.101 with username : root password : couchbase ssh_key: 2016-11-04 12:34:00 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.101 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [fts_base.are_index_files_deleted_from_disk] 0 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.151.102 with username : root password : couchbase ssh_key: 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.151.102 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [fts_base.are_index_files_deleted_from_disk] 0 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [fts_base.delete] Validated: all index files for default_index deleted from disk 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 10.111.151.101 2016-11-04 12:34:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 10.111.151.101 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 10.111.151.101 existing buckets : [] 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2016-11-04 12:34:03 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2016-11-04 12:34:23 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.0177748203 seconds 2016-11-04 12:34:23 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.151.101:8091 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.151.101:8091 is running 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.151.102 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.151.102:8091 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force 2016-11-04 12:34:33 | INFO | MainProcess | test_thread | [fts_base.tearDown] ==== FTSbasetests cleanup is finished for test #1 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 0 , fail 1 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_1 Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_2 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=salary,expected=emp10000042,emp10000001,emp10000071,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'salary', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 2, 'expected': 'emp10000042,emp10000001,emp10000071', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_2', 'spec': 'test'} [2016-11-04 12:34:33,623] - [fts_base:2530] INFO - ==== FTSbasetests setup is started for test #2 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:34:33,630] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:34:33,636] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root] [2016-11-04 12:34:33,641] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.101 [2016-11-04 12:34:33,655] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:34:33,658] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:34:33,658] - [fts_base:1444] INFO - Initializing Cluster ... [2016-11-04 12:34:34,636] - [rest_client:887] INFO - pools/default params : memoryQuota=1539 [2016-11-04 12:34:34,640] - [rest_client:924] INFO - settings/indexes params : storageMode=forestdb [2016-11-04 12:34:34,646] - [rest_client:803] INFO - settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 [2016-11-04 12:34:34,660] - [fts_base:1460] INFO - 10.111.151.102 will be configured with services fts [2016-11-04 12:34:35,665] - [task:409] INFO - adding node 10.111.151.102:8091 to cluster [2016-11-04 12:34:35,666] - [rest_client:1082] INFO - adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 [2016-11-04 12:34:38,385] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:34:38,397] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:34:38,402] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:34:48,445] - [task:532] INFO - rebalancing was completed with progress: 100% in 10.0477898121 sec [2016-11-04 12:34:49,461] - [rest_client:1923] INFO - http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-11-04 12:34:49,469] - [rest_client:1945] INFO - 0.01 seconds to create bucket default [2016-11-04 12:34:49,469] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 10.111.151.101 to accept set ops [2016-11-04 12:34:50,879] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:34:51,194] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:34:52,694] - [task:293] INFO - bucket 'default' was created with per node RAM quota: 1039 [2016-11-04 12:34:52,715] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:34:52,716] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:34:52,823] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:34:53,258] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:34:53,379] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:34:53,379] - [fts_base:3034] INFO - 0 [2016-11-04 12:34:53,380] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:34:53,488] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:34:53,489] - [fts_base:3034] INFO - 0 [2016-11-04 12:34:53,491] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:34:53,600] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:34:53,941] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:34:54,029] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:34:54,029] - [fts_base:3034] INFO - 0 [2016-11-04 12:34:54,030] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:34:54,086] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:34:54,087] - [fts_base:3034] INFO - 0 [2016-11-04 12:34:54,089] - [fts_base:2750] INFO - {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} [2016-11-04 12:34:54,089] - [fts_base:2539] INFO - ==== FTSbasetests setup is finished for test #2 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:34:54,122] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:34:54,771] - [fts_base:3451] INFO - Loading phase complete! [2016-11-04 12:34:54,815] - [fts_base:838] INFO - Checking if index already exists ... [2016-11-04 12:34:54,824] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:34:54,826] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:34:54,826] - [fts_base:846] INFO - Creating fulltext-index default_index on 10.111.151.101 [2016-11-04 12:34:54,826] - [rest_client:2249] INFO - {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-11-04 12:34:54,847] - [rest_client:2258] INFO - Index default_index created [2016-11-04 12:34:54,847] - [fts_base:3144] INFO - Validating index distribution for default_index ... [2016-11-04 12:34:54,870] - [fts_base:3055] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-11-04 12:34:59,904] - [fts_base:3162] INFO - Validated: Number of PIndexes = 32 [2016-11-04 12:34:59,915] - [fts_base:3174] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-11-04 12:34:59,916] - [fts_base:3190] INFO - Validated: pIndexes are distributed across [u'011ccecb1432b991dcabff4a0da485c9', u'8cffca21ccf79caf1a73da9e66d3ff4f'] [2016-11-04 12:34:59,916] - [fts_base:3196] INFO - Expecting num of partitions in each node in range 480-544 [2016-11-04 12:34:59,916] - [fts_base:3211] INFO - Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions [2016-11-04 12:34:59,916] - [fts_base:3211] INFO - Validated: Node 8cffca21ccf79caf1a73da9e66d3ff4f houses 16 pindexes which serve 512 partitions [2016-11-04 12:34:59,945] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:34:59,945] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:34:59,968] - [fts_base:854] INFO - Updating fulltext-index default_index on 10.111.151.102 [2016-11-04 12:34:59,969] - [rest_client:2265] INFO - { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "7bcac687621b55ac" } [2016-11-04 12:34:59,991] - [rest_client:2274] INFO - Index/alias default_index updated [2016-11-04 12:34:59,992] - [fts_base:3055] INFO - sleep for 5 secs. ... [2016-11-04 12:35:05,029] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:35:05,029] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:35:05,036] - [fts_base:1694] INFO - Running query {"sort": ["salary"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: [2016-11-04 12:35:05,067] - [stable_topology_fts:858] INFO - Hits: 3 [2016-11-04 12:35:05,067] - [stable_topology_fts:859] INFO - Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_61ac9b9d56b03392_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_61ac9b9d56b03392_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_61ac9b9d56b03392_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}] [2016-11-04 12:35:05,067] - [fts_base:1227] INFO - Actual docs returned : [u'emp10000001', u'emp10000042', u'emp10000071'] [2016-11-04 12:35:05,067] - [fts_base:1228] INFO - Expected docs : ['emp10000042', 'emp10000001', 'emp10000071'] [2016-11-04 12:35:05,068] - [stable_topology_fts:867] ERROR - Testcase failed. Actual results do not match expected. [2016-11-04 12:35:05,073] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:35:05,074] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:35:05,177] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:35:05,513] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:05,655] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:05,655] - [fts_base:3034] INFO - 0 [2016-11-04 12:35:05,655] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.101 :0, now :0 [2016-11-04 12:35:05,655] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:05,766] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:05,767] - [fts_base:3034] INFO - 0 [2016-11-04 12:35:05,767] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 [2016-11-04 12:35:05,769] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:35:05,877] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:35:06,212] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:06,304] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:06,305] - [fts_base:3034] INFO - 0 [2016-11-04 12:35:06,305] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.102 :0, now :0 [2016-11-04 12:35:06,305] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:06,390] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:06,391] - [fts_base:3034] INFO - 0 [2016-11-04 12:35:06,391] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 [2016-11-04 12:35:06,392] - [fts_base:2600] INFO - ==== FTSbasetests cleanup is started for test #2 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:35:06,403] - [fts_base:862] INFO - Deleting fulltext-index default_index on 10.111.151.102 [2016-11-04 12:35:06,453] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:35:06,556] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:35:06,900] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:35:06,984] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:06,984] - [fts_base:1407] INFO - 0 [2016-11-04 12:35:06,992] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:35:07,096] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:35:07,659] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:35:07,739] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:07,740] - [fts_base:1407] INFO - 0 [2016-11-04 12:35:07,864] - [fts_base:869] INFO - Validated: all index files for default_index deleted from disk [2016-11-04 12:35:07,864] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:35:07,871] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] [2016-11-04 12:35:07,883] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 10.111.151.101 [2016-11-04 12:35:07,883] - [bucket_helper:140] INFO - remove bucket default ... [2016-11-04 12:35:09,555] - [bucket_helper:154] INFO - deleted bucket : default from 10.111.151.101 [2016-11-04 12:35:09,556] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-11-04 12:35:09,560] - [rest_client:134] INFO - node 10.111.151.101 existing buckets : [] [2016-11-04 12:35:09,574] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes [2016-11-04 12:35:09,576] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:35:09,585] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:35:09,589] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:35:29,604] - [rest_client:1390] INFO - rebalance progress took 20.0195639133 seconds [2016-11-04 12:35:29,605] - [rest_client:1391] INFO - sleep for 10 seconds after rebalance... [2016-11-04 12:35:39,615] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] [2016-11-04 12:35:39,619] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:35:39,622] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:35:39,628] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.102 [2016-11-04 12:35:39,640] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.102:8091 [2016-11-04 12:35:39,643] - [cluster_helper:80] INFO - ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force [2016-11-04 12:35:39,643] - [fts_base:2606] INFO - ==== FTSbasetests cleanup is finished for test #2 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 0 , fail 2 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_2 Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_3 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=dept,expected=emp10000042,emp10000001,emp10000071,tolerance=emp10000001,emp10000071,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'dept', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 3, 'expected': 'emp10000042,emp10000001,emp10000071', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_3', 'tolerance': 'emp10000001,emp10000071', 'spec': 'test'} [2016-11-04 12:35:39,647] - [fts_base:2530] INFO - ==== FTSbasetests setup is started for test #3 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:35:39,655] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:35:39,661] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root] [2016-11-04 12:35:39,666] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.101 [2016-11-04 12:35:39,678] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:35:39,681] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:35:39,681] - [fts_base:1444] INFO - Initializing Cluster ... [2016-11-04 12:35:40,659] - [rest_client:887] INFO - pools/default params : memoryQuota=1539 [2016-11-04 12:35:40,663] - [rest_client:924] INFO - settings/indexes params : storageMode=forestdb [2016-11-04 12:35:40,667] - [rest_client:803] INFO - settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 [2016-11-04 12:35:40,686] - [fts_base:1460] INFO - 10.111.151.102 will be configured with services fts [2016-11-04 12:35:41,690] - [task:409] INFO - adding node 10.111.151.102:8091 to cluster [2016-11-04 12:35:41,691] - [rest_client:1082] INFO - adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 [2016-11-04 12:35:44,540] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:35:44,548] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:35:44,553] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:35:54,596] - [task:532] INFO - rebalancing was completed with progress: 100% in 10.0472900867 sec [2016-11-04 12:35:55,612] - [rest_client:1923] INFO - http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-11-04 12:35:55,619] - [rest_client:1945] INFO - 0.01 seconds to create bucket default [2016-11-04 12:35:55,620] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 10.111.151.101 to accept set ops [2016-11-04 12:35:57,034] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:35:57,349] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:35:59,102] - [task:293] INFO - bucket 'default' was created with per node RAM quota: 1039 [2016-11-04 12:35:59,234] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:35:59,235] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:35:59,326] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:35:59,702] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:59,824] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:59,824] - [fts_base:3034] INFO - 0 [2016-11-04 12:35:59,825] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:35:59,901] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:35:59,902] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:00,028] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:36:00,136] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:36:00,470] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:00,560] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:00,560] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:00,561] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:00,648] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:00,648] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:00,773] - [fts_base:2750] INFO - {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} [2016-11-04 12:36:00,773] - [fts_base:2539] INFO - ==== FTSbasetests setup is finished for test #3 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:36:00,811] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:36:01,147] - [fts_base:3451] INFO - Loading phase complete! [2016-11-04 12:36:01,228] - [fts_base:838] INFO - Checking if index already exists ... [2016-11-04 12:36:01,239] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:36:01,241] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:36:01,242] - [fts_base:846] INFO - Creating fulltext-index default_index on 10.111.151.101 [2016-11-04 12:36:01,242] - [rest_client:2249] INFO - {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-11-04 12:36:01,262] - [rest_client:2258] INFO - Index default_index created [2016-11-04 12:36:01,263] - [fts_base:3144] INFO - Validating index distribution for default_index ... [2016-11-04 12:36:01,286] - [fts_base:3055] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-11-04 12:36:06,320] - [fts_base:3162] INFO - Validated: Number of PIndexes = 32 [2016-11-04 12:36:06,326] - [fts_base:3174] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-11-04 12:36:06,326] - [fts_base:3190] INFO - Validated: pIndexes are distributed across [u'011ccecb1432b991dcabff4a0da485c9', u'b8db7e12390572bd9b5a0bb831a59cc1'] [2016-11-04 12:36:06,326] - [fts_base:3196] INFO - Expecting num of partitions in each node in range 480-544 [2016-11-04 12:36:06,326] - [fts_base:3211] INFO - Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions [2016-11-04 12:36:06,327] - [fts_base:3211] INFO - Validated: Node b8db7e12390572bd9b5a0bb831a59cc1 houses 16 pindexes which serve 512 partitions [2016-11-04 12:36:06,356] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:36:06,356] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:36:06,378] - [fts_base:854] INFO - Updating fulltext-index default_index on 10.111.151.102 [2016-11-04 12:36:06,379] - [rest_client:2265] INFO - { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "7a4fa6082170cd0a" } [2016-11-04 12:36:06,398] - [rest_client:2274] INFO - Index/alias default_index updated [2016-11-04 12:36:06,398] - [fts_base:3055] INFO - sleep for 5 secs. ... [2016-11-04 12:36:11,433] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:36:11,434] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:36:11,440] - [fts_base:1694] INFO - Running query {"sort": ["dept"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: [2016-11-04 12:36:11,475] - [stable_topology_fts:858] INFO - Hits: 3 [2016-11-04 12:36:11,476] - [stable_topology_fts:859] INFO - Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_4e4f812bda005b70_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_4e4f812bda005b70_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_4e4f812bda005b70_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}] [2016-11-04 12:36:11,476] - [fts_base:1223] INFO - Actual docs returned : [u'emp10000071', u'emp10000001', u'emp10000042'] [2016-11-04 12:36:11,476] - [fts_base:1224] INFO - Expected docs : ['emp10000042', 'emp10000071', 'emp10000001'] [2016-11-04 12:36:11,476] - [stable_topology_fts:867] ERROR - Testcase failed. Actual results do not match expected. [2016-11-04 12:36:11,483] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:36:11,483] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:36:11,587] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:36:11,978] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:12,102] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:12,102] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:12,102] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.101 :0, now :0 [2016-11-04 12:36:12,103] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:12,213] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:12,213] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:12,214] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 [2016-11-04 12:36:12,216] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:36:12,320] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:36:12,691] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:12,782] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:12,783] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:12,783] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.102 :0, now :0 [2016-11-04 12:36:12,783] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:36:12,841] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:12,842] - [fts_base:3034] INFO - 0 [2016-11-04 12:36:12,842] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 [2016-11-04 12:36:12,844] - [fts_base:2600] INFO - ==== FTSbasetests cleanup is started for test #3 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:36:12,854] - [fts_base:862] INFO - Deleting fulltext-index default_index on 10.111.151.102 [2016-11-04 12:36:12,908] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:36:13,022] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:36:13,368] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:36:13,452] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:13,452] - [fts_base:1407] INFO - 0 [2016-11-04 12:36:13,460] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:36:13,567] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:36:14,031] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:36:14,110] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:36:14,111] - [fts_base:1407] INFO - 0 [2016-11-04 12:36:14,112] - [fts_base:869] INFO - Validated: all index files for default_index deleted from disk [2016-11-04 12:36:14,112] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:36:14,118] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] [2016-11-04 12:36:14,130] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 10.111.151.101 [2016-11-04 12:36:14,130] - [bucket_helper:140] INFO - remove bucket default ... [2016-11-04 12:36:15,804] - [bucket_helper:154] INFO - deleted bucket : default from 10.111.151.101 [2016-11-04 12:36:15,804] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-11-04 12:36:15,809] - [rest_client:134] INFO - node 10.111.151.101 existing buckets : [] [2016-11-04 12:36:15,824] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes [2016-11-04 12:36:15,826] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:36:15,835] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:36:15,840] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:36:35,854] - [rest_client:1390] INFO - rebalance progress took 20.01906991 seconds [2016-11-04 12:36:35,854] - [rest_client:1391] INFO - sleep for 10 seconds after rebalance... [2016-11-04 12:36:45,866] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] [2016-11-04 12:36:45,870] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:36:45,873] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:36:45,878] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.102 [2016-11-04 12:36:45,894] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.102:8091 [2016-11-04 12:36:45,897] - [cluster_helper:80] INFO - ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force [2016-11-04 12:36:45,897] - [fts_base:2606] INFO - ==== FTSbasetests cleanup is finished for test #3 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 0 , fail 3 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_3 Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_4 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=dept,expected=emp10000042,emp10000001,emp10000071,tolerance=emp10000001,emp10000071,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'dept', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 4, 'expected': 'emp10000042,emp10000001,emp10000071', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_4', 'tolerance': 'emp10000001,emp10000071', 'spec': 'test'} [2016-11-04 12:36:45,902] - [fts_base:2530] INFO - ==== FTSbasetests setup is started for test #4 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:36:45,909] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:36:45,914] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root] [2016-11-04 12:36:45,919] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.101 [2016-11-04 12:36:45,935] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:36:45,938] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:36:45,938] - [fts_base:1444] INFO - Initializing Cluster ... [2016-11-04 12:36:46,915] - [rest_client:887] INFO - pools/default params : memoryQuota=1539 [2016-11-04 12:36:46,918] - [rest_client:924] INFO - settings/indexes params : storageMode=forestdb [2016-11-04 12:36:46,923] - [rest_client:803] INFO - settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 [2016-11-04 12:36:46,942] - [fts_base:1460] INFO - 10.111.151.102 will be configured with services fts [2016-11-04 12:36:47,947] - [task:409] INFO - adding node 10.111.151.102:8091 to cluster [2016-11-04 12:36:47,947] - [rest_client:1082] INFO - adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 [2016-11-04 12:36:50,990] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:36:51,000] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:36:51,005] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:37:01,058] - [task:532] INFO - rebalancing was completed with progress: 100% in 10.057420969 sec [2016-11-04 12:37:02,076] - [rest_client:1923] INFO - http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-11-04 12:37:02,087] - [rest_client:1945] INFO - 0.01 seconds to create bucket default [2016-11-04 12:37:02,087] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 10.111.151.101 to accept set ops [2016-11-04 12:37:03,527] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:37:03,829] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:37:05,098] - [task:293] INFO - bucket 'default' was created with per node RAM quota: 1039 [2016-11-04 12:37:05,135] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:37:05,136] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:37:05,243] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:37:05,612] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:05,735] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:05,736] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:05,736] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:05,813] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:05,813] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:05,816] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:37:05,923] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:37:06,293] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:06,383] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:06,383] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:06,384] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:06,468] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:06,469] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:06,471] - [fts_base:2750] INFO - {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} [2016-11-04 12:37:06,471] - [fts_base:2539] INFO - ==== FTSbasetests setup is finished for test #4 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:37:06,506] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:37:07,141] - [fts_base:3451] INFO - Loading phase complete! [2016-11-04 12:37:07,223] - [fts_base:838] INFO - Checking if index already exists ... [2016-11-04 12:37:07,236] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:37:07,238] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:37:07,238] - [fts_base:846] INFO - Creating fulltext-index default_index on 10.111.151.101 [2016-11-04 12:37:07,239] - [rest_client:2249] INFO - {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-11-04 12:37:07,257] - [rest_client:2258] INFO - Index default_index created [2016-11-04 12:37:07,257] - [fts_base:3144] INFO - Validating index distribution for default_index ... [2016-11-04 12:37:07,280] - [fts_base:3055] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-11-04 12:37:12,325] - [fts_base:3162] INFO - Validated: Number of PIndexes = 32 [2016-11-04 12:37:12,330] - [fts_base:3174] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-11-04 12:37:12,331] - [fts_base:3190] INFO - Validated: pIndexes are distributed across [u'5676fc7a35f01e8ab25beca9ca6e9174', u'011ccecb1432b991dcabff4a0da485c9'] [2016-11-04 12:37:12,331] - [fts_base:3196] INFO - Expecting num of partitions in each node in range 480-544 [2016-11-04 12:37:12,331] - [fts_base:3211] INFO - Validated: Node 5676fc7a35f01e8ab25beca9ca6e9174 houses 16 pindexes which serve 512 partitions [2016-11-04 12:37:12,331] - [fts_base:3211] INFO - Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions [2016-11-04 12:37:12,360] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:37:12,360] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:37:12,384] - [fts_base:854] INFO - Updating fulltext-index default_index on 10.111.151.102 [2016-11-04 12:37:12,384] - [rest_client:2265] INFO - { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "51695ad0efeef61a" } [2016-11-04 12:37:12,404] - [rest_client:2274] INFO - Index/alias default_index updated [2016-11-04 12:37:12,405] - [fts_base:3055] INFO - sleep for 5 secs. ... [2016-11-04 12:37:17,442] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:37:17,442] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:37:17,449] - [fts_base:1694] INFO - Running query {"sort": ["dept"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: [2016-11-04 12:37:17,473] - [stable_topology_fts:858] INFO - Hits: 3 [2016-11-04 12:37:17,473] - [stable_topology_fts:859] INFO - Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_16f56273c2ef4261_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_16f56273c2ef4261_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_16f56273c2ef4261_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}] [2016-11-04 12:37:17,479] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:37:17,480] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:37:17,586] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:37:17,952] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:18,080] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:18,081] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:18,081] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.101 :0, now :0 [2016-11-04 12:37:18,081] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:18,164] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:18,164] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:18,164] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 [2016-11-04 12:37:18,167] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:37:18,266] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:37:18,635] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:18,727] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:18,727] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:18,727] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.102 :0, now :0 [2016-11-04 12:37:18,728] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:37:18,817] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:18,818] - [fts_base:3034] INFO - 0 [2016-11-04 12:37:18,818] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 [2016-11-04 12:37:18,819] - [fts_base:2600] INFO - ==== FTSbasetests cleanup is started for test #4 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:37:18,830] - [fts_base:862] INFO - Deleting fulltext-index default_index on 10.111.151.102 [2016-11-04 12:37:18,876] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:37:18,985] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:37:19,325] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:37:19,408] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:19,409] - [fts_base:1407] INFO - 0 [2016-11-04 12:37:19,416] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:37:19,519] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:37:19,991] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:37:20,075] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:37:20,075] - [fts_base:1407] INFO - 0 [2016-11-04 12:37:20,077] - [fts_base:869] INFO - Validated: all index files for default_index deleted from disk [2016-11-04 12:37:20,077] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:37:20,083] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] [2016-11-04 12:37:20,094] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 10.111.151.101 [2016-11-04 12:37:20,095] - [bucket_helper:140] INFO - remove bucket default ... [2016-11-04 12:37:21,755] - [bucket_helper:154] INFO - deleted bucket : default from 10.111.151.101 [2016-11-04 12:37:21,756] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-11-04 12:37:21,760] - [rest_client:134] INFO - node 10.111.151.101 existing buckets : [] [2016-11-04 12:37:21,773] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes [2016-11-04 12:37:21,776] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:37:21,784] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:37:21,789] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:37:41,803] - [rest_client:1390] INFO - rebalance progress took 20.0190639496 seconds [2016-11-04 12:37:41,804] - [rest_client:1391] INFO - sleep for 10 seconds after rebalance... [2016-11-04 12:37:51,811] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] [2016-11-04 12:37:51,815] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:37:51,818] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:37:51,823] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.102 [2016-11-04 12:37:51,837] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.102:8091 [2016-11-04 12:37:51,840] - [cluster_helper:80] INFO - ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force [2016-11-04 12:37:51,840] - [fts_base:2606] INFO - ==== FTSbasetests cleanup is finished for test #4 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 1 , fail 3 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_4 Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_5 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=join_date,expected=emp10000001,emp10000071,emp10000042,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'join_date', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 5, 'expected': 'emp10000001,emp10000071,emp10000042', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_5', 'spec': 'test'} [2016-11-04 12:37:51,844] - [fts_base:2530] INFO - ==== FTSbasetests setup is started for test #5 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:37:51,851] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:37:51,856] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root] [2016-11-04 12:37:51,861] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.101 [2016-11-04 12:37:51,874] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:37:51,877] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:37:51,877] - [fts_base:1444] INFO - Initializing Cluster ... [2016-11-04 12:37:52,855] - [rest_client:887] INFO - pools/default params : memoryQuota=1539 [2016-11-04 12:37:52,859] - [rest_client:924] INFO - settings/indexes params : storageMode=forestdb [2016-11-04 12:37:52,865] - [rest_client:803] INFO - settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 [2016-11-04 12:37:52,883] - [fts_base:1460] INFO - 10.111.151.102 will be configured with services fts [2016-11-04 12:37:53,887] - [task:409] INFO - adding node 10.111.151.102:8091 to cluster [2016-11-04 12:37:53,888] - [rest_client:1082] INFO - adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 [2016-11-04 12:37:56,878] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:37:56,888] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:37:56,893] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:38:06,947] - [task:532] INFO - rebalancing was completed with progress: 100% in 10.0589950085 sec [2016-11-04 12:38:07,963] - [rest_client:1923] INFO - http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-11-04 12:38:07,970] - [rest_client:1945] INFO - 0.01 seconds to create bucket default [2016-11-04 12:38:07,970] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 10.111.151.101 to accept set ops [2016-11-04 12:38:09,397] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:38:09,708] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:38:11,055] - [task:293] INFO - bucket 'default' was created with per node RAM quota: 1039 [2016-11-04 12:38:11,065] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:38:11,066] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:38:11,169] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:38:11,507] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:11,636] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:11,636] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:11,636] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:11,723] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:11,723] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:11,725] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:38:11,827] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:38:12,198] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:12,290] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:12,290] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:12,290] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:12,350] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:12,350] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:12,352] - [fts_base:2750] INFO - {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} [2016-11-04 12:38:12,352] - [fts_base:2539] INFO - ==== FTSbasetests setup is finished for test #5 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:38:12,399] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:38:13,105] - [fts_base:3451] INFO - Loading phase complete! [2016-11-04 12:38:13,189] - [fts_base:838] INFO - Checking if index already exists ... [2016-11-04 12:38:13,200] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:38:13,202] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:38:13,202] - [fts_base:846] INFO - Creating fulltext-index default_index on 10.111.151.101 [2016-11-04 12:38:13,203] - [rest_client:2249] INFO - {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-11-04 12:38:13,222] - [rest_client:2258] INFO - Index default_index created [2016-11-04 12:38:13,223] - [fts_base:3144] INFO - Validating index distribution for default_index ... [2016-11-04 12:38:13,248] - [fts_base:3055] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-11-04 12:38:18,281] - [fts_base:3162] INFO - Validated: Number of PIndexes = 32 [2016-11-04 12:38:18,287] - [fts_base:3174] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-11-04 12:38:18,287] - [fts_base:3190] INFO - Validated: pIndexes are distributed across [u'9f00f433315aa0a6dc65c3027f85e53b', u'011ccecb1432b991dcabff4a0da485c9'] [2016-11-04 12:38:18,287] - [fts_base:3196] INFO - Expecting num of partitions in each node in range 480-544 [2016-11-04 12:38:18,288] - [fts_base:3211] INFO - Validated: Node 9f00f433315aa0a6dc65c3027f85e53b houses 16 pindexes which serve 512 partitions [2016-11-04 12:38:18,288] - [fts_base:3211] INFO - Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions [2016-11-04 12:38:18,316] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:38:18,316] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:38:18,339] - [fts_base:854] INFO - Updating fulltext-index default_index on 10.111.151.102 [2016-11-04 12:38:18,339] - [rest_client:2265] INFO - { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "79216f42c72a6025" } [2016-11-04 12:38:18,359] - [rest_client:2274] INFO - Index/alias default_index updated [2016-11-04 12:38:18,360] - [fts_base:3055] INFO - sleep for 5 secs. ... [2016-11-04 12:38:23,400] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:38:23,400] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:38:23,407] - [fts_base:1694] INFO - Running query {"sort": ["join_date"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: [2016-11-04 12:38:23,442] - [stable_topology_fts:858] INFO - Hits: 3 [2016-11-04 12:38:23,443] - [stable_topology_fts:859] INFO - Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_348bf37d01ce78db_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_348bf37d01ce78db_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_348bf37d01ce78db_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}] [2016-11-04 12:38:23,443] - [fts_base:1227] INFO - Actual docs returned : [u'emp10000071', u'emp10000001', u'emp10000042'] [2016-11-04 12:38:23,443] - [fts_base:1228] INFO - Expected docs : ['emp10000001', 'emp10000071', 'emp10000042'] [2016-11-04 12:38:23,443] - [stable_topology_fts:867] ERROR - Testcase failed. Actual results do not match expected. [2016-11-04 12:38:23,449] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:38:23,450] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:38:23,557] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:38:23,909] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:24,031] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:24,031] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:24,031] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.101 :0, now :0 [2016-11-04 12:38:24,032] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:24,112] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:24,112] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:24,113] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 [2016-11-04 12:38:24,115] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:38:24,223] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:38:24,561] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:24,650] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:24,651] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:24,651] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.102 :0, now :0 [2016-11-04 12:38:24,651] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:38:24,742] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:24,742] - [fts_base:3034] INFO - 0 [2016-11-04 12:38:24,743] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 [2016-11-04 12:38:24,864] - [fts_base:2600] INFO - ==== FTSbasetests cleanup is started for test #5 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:38:24,877] - [fts_base:862] INFO - Deleting fulltext-index default_index on 10.111.151.102 [2016-11-04 12:38:24,919] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:38:25,040] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:38:25,415] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:38:25,502] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:25,502] - [fts_base:1407] INFO - 0 [2016-11-04 12:38:25,509] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:38:25,614] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:38:26,076] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:38:26,159] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:38:26,160] - [fts_base:1407] INFO - 0 [2016-11-04 12:38:26,161] - [fts_base:869] INFO - Validated: all index files for default_index deleted from disk [2016-11-04 12:38:26,162] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:38:26,168] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] [2016-11-04 12:38:26,179] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 10.111.151.101 [2016-11-04 12:38:26,179] - [bucket_helper:140] INFO - remove bucket default ... [2016-11-04 12:38:27,838] - [bucket_helper:154] INFO - deleted bucket : default from 10.111.151.101 [2016-11-04 12:38:27,838] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-11-04 12:38:27,842] - [rest_client:134] INFO - node 10.111.151.101 existing buckets : [] [2016-11-04 12:38:27,855] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes [2016-11-04 12:38:27,858] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:38:27,865] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:38:27,869] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:38:47,884] - [rest_client:1390] INFO - rebalance progress took 20.0192639828 seconds [2016-11-04 12:38:47,885] - [rest_client:1391] INFO - sleep for 10 seconds after rebalance... [2016-11-04 12:38:57,893] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] [2016-11-04 12:38:57,897] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:38:57,901] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:38:57,907] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.102 [2016-11-04 12:38:57,918] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.102:8091 [2016-11-04 12:38:57,921] - [cluster_helper:80] INFO - ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force [2016-11-04 12:38:57,921] - [fts_base:2606] INFO - ==== FTSbasetests cleanup is finished for test #5 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 1 , fail 4 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_5 Logs will be stored at /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_6 ./testrunner -i fts.ini -t fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields,items=100,sort_fields=join_date,expected=emp10000001,emp10000071,emp10000042,cluster=D+F,F Test Input params: {'items': '100', 'sort_fields': 'join_date', 'conf_file': 'conf/fts/test.conf', 'num_nodes': 2, 'cluster_name': 'fts', 'cluster': 'D+F,F', 'ini': 'fts.ini', 'case_number': 6, 'expected': 'emp10000001,emp10000071,emp10000042', 'logs_folder': '/Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_6', 'spec': 'test'} [2016-11-04 12:38:57,925] - [fts_base:2530] INFO - ==== FTSbasetests setup is started for test #6 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:38:57,932] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:38:57,938] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root] [2016-11-04 12:38:57,943] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.101 [2016-11-04 12:38:57,957] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:38:57,960] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:38:57,960] - [fts_base:1444] INFO - Initializing Cluster ... [2016-11-04 12:38:58,938] - [rest_client:887] INFO - pools/default params : memoryQuota=1539 [2016-11-04 12:38:58,941] - [rest_client:924] INFO - settings/indexes params : storageMode=forestdb [2016-11-04 12:38:58,946] - [rest_client:803] INFO - settings/web params on 10.111.151.101:8091:username=Administrator&password=password&port=8091 [2016-11-04 12:38:58,966] - [fts_base:1460] INFO - 10.111.151.102 will be configured with services fts [2016-11-04 12:38:59,971] - [task:409] INFO - adding node 10.111.151.102:8091 to cluster [2016-11-04 12:38:59,971] - [rest_client:1082] INFO - adding remote node @10.111.151.102:8091 to this cluster @10.111.151.101:8091 [2016-11-04 12:39:03,262] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:39:03,271] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:39:03,275] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:39:13,317] - [task:532] INFO - rebalancing was completed with progress: 100% in 10.0455620289 sec [2016-11-04 12:39:14,331] - [rest_client:1923] INFO - http://10.111.151.101:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1039&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-11-04 12:39:14,338] - [rest_client:1945] INFO - 0.01 seconds to create bucket default [2016-11-04 12:39:14,338] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 10.111.151.101 to accept set ops [2016-11-04 12:39:15,821] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:39:16,169] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:39:17,522] - [task:293] INFO - bucket 'default' was created with per node RAM quota: 1039 [2016-11-04 12:39:17,579] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:39:17,580] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:39:17,678] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:39:18,014] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:18,136] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:18,137] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:18,137] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:18,246] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:18,246] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:18,249] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:39:18,351] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:39:18,820] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:18,915] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:18,915] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:18,915] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:19,005] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:19,006] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:19,008] - [fts_base:2750] INFO - {'10.111.151.102': {'crash in forestdb': 0, 'panic:': 0}, '10.111.151.101': {'crash in forestdb': 0, 'panic:': 0}} [2016-11-04 12:39:19,008] - [fts_base:2539] INFO - ==== FTSbasetests setup is finished for test #6 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:39:19,048] - [data_helper:295] INFO - creating direct client 10.111.151.101:11210 default [2016-11-04 12:39:19,564] - [fts_base:3451] INFO - Loading phase complete! [2016-11-04 12:39:19,617] - [fts_base:838] INFO - Checking if index already exists ... [2016-11-04 12:39:19,669] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:39:19,671] - [rest_client:786] ERROR - http://10.111.151.101:8094/api/index/default_index error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found rest_auth: preparePerm, err: index not found [2016-11-04 12:39:19,672] - [fts_base:846] INFO - Creating fulltext-index default_index on 10.111.151.101 [2016-11-04 12:39:19,672] - [rest_client:2249] INFO - {"params": {}, "name": "default_index", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""} [2016-11-04 12:39:19,692] - [rest_client:2258] INFO - Index default_index created [2016-11-04 12:39:19,692] - [fts_base:3144] INFO - Validating index distribution for default_index ... [2016-11-04 12:39:19,715] - [fts_base:3055] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ... [2016-11-04 12:39:24,750] - [fts_base:3162] INFO - Validated: Number of PIndexes = 32 [2016-11-04 12:39:24,757] - [fts_base:3174] INFO - Validated: Every pIndex serves 32 partitions or lesser [2016-11-04 12:39:24,758] - [fts_base:3190] INFO - Validated: pIndexes are distributed across [u'011ccecb1432b991dcabff4a0da485c9', u'e8fba0dd4989ff673f807b79c509393d'] [2016-11-04 12:39:24,758] - [fts_base:3196] INFO - Expecting num of partitions in each node in range 480-544 [2016-11-04 12:39:24,758] - [fts_base:3211] INFO - Validated: Node 011ccecb1432b991dcabff4a0da485c9 houses 16 pindexes which serve 512 partitions [2016-11-04 12:39:24,758] - [fts_base:3211] INFO - Validated: Node e8fba0dd4989ff673f807b79c509393d houses 16 pindexes which serve 512 partitions [2016-11-04 12:39:24,788] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:39:24,788] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:39:24,813] - [fts_base:854] INFO - Updating fulltext-index default_index on 10.111.151.102 [2016-11-04 12:39:24,813] - [rest_client:2265] INFO - { "params": { "mapping": { "default_mapping": { "enabled": true, "dynamic": false, "properties": { "name": { "enabled": true, "dynamic": false, "fields": [ { "index": true, "store": true, "name": "name", "include_in_all": true, "type": "text", "display_order": "0", "include_term_vectors": true, "analyzer": "en" } ] } } } } }, "name": "default_index", "planParams": { "hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32 }, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": { "authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0 }, "uuid": "4a73715598fea1fe" } [2016-11-04 12:39:24,835] - [rest_client:2274] INFO - Index/alias default_index updated [2016-11-04 12:39:24,836] - [fts_base:3055] INFO - sleep for 5 secs. ... [2016-11-04 12:39:29,871] - [fts_base:3077] INFO - Docs in bucket = 100, docs in FTS index 'default_index': 100 [2016-11-04 12:39:29,872] - [fts_base:3097] INFO - FTS indexed 100 docs in 0.0 mins [2016-11-04 12:39:29,878] - [fts_base:1694] INFO - Running query {"sort": ["join_date"], "from": 0, "indexName": "default_index", "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Safiya"}, {"field": "name", "match": "Palmer"}]}, "size": 10000000} on node: 10.111.151.101: [2016-11-04 12:39:29,904] - [stable_topology_fts:858] INFO - Hits: 3 [2016-11-04 12:39:29,904] - [stable_topology_fts:859] INFO - Doc IDs: [{u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_358f1837ddfc5af0_b7ff6b68', u'score': 1.9162906990782362, u'id': u'emp10000001', u'locations': {u'name': {u'palmer': [{u'start': 7, u'end': 13, u'pos': 2, u'array_positions': None}], u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_358f1837ddfc5af0_d6109bff', u'score': 0.3464004250432601, u'id': u'emp10000042', u'locations': {u'name': {u'safiya': [{u'start': 0, u'end': 6, u'pos': 1, u'array_positions': None}]}}}, {u'sort': [u'\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd'], u'index': u'default_index_358f1837ddfc5af0_596cc6a1', u'score': 0.3464004250432601, u'id': u'emp10000071', u'locations': {u'name': {u'palmer': [{u'start': 10, u'end': 16, u'pos': 2, u'array_positions': None}]}}}] [2016-11-04 12:39:29,905] - [fts_base:1227] INFO - Actual docs returned : [u'emp10000001', u'emp10000042', u'emp10000071'] [2016-11-04 12:39:29,905] - [fts_base:1228] INFO - Expected docs : ['emp10000001', 'emp10000071', 'emp10000042'] [2016-11-04 12:39:29,905] - [stable_topology_fts:867] ERROR - Testcase failed. Actual results do not match expected. [2016-11-04 12:39:29,914] - [rest_client:1316] INFO - /diag/eval status on 10.111.151.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2016-11-04 12:39:29,915] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:39:30,022] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:39:30,393] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:30,516] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:30,516] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:30,516] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.101 :0, now :0 [2016-11-04 12:39:30,517] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:30,627] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:30,627] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:30,627] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.101 :0, now :0 [2016-11-04 12:39:30,630] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:39:30,739] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:39:31,106] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:31,196] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:31,197] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:31,197] - [fts_base:3042] INFO - Initial 'panic:' count on 10.111.151.102 :0, now :0 [2016-11-04 12:39:31,198] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l [2016-11-04 12:39:31,285] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:31,285] - [fts_base:3034] INFO - 0 [2016-11-04 12:39:31,286] - [fts_base:3042] INFO - Initial 'crash in forestdb' count on 10.111.151.102 :0, now :0 [2016-11-04 12:39:31,413] - [fts_base:2600] INFO - ==== FTSbasetests cleanup is started for test #6 test_sorting_of_results_on_non_indexed_fields ==== [2016-11-04 12:39:31,425] - [fts_base:862] INFO - Deleting fulltext-index default_index on 10.111.151.102 [2016-11-04 12:39:31,474] - [remote_util:172] INFO - connecting to 10.111.151.101 with username : root password : couchbase ssh_key: [2016-11-04 12:39:31,577] - [remote_util:206] INFO - Connected to 10.111.151.101 [2016-11-04 12:39:31,930] - [remote_util:2383] INFO - running command.raw on 10.111.151.101: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:39:32,016] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:32,017] - [fts_base:1407] INFO - 0 [2016-11-04 12:39:32,024] - [remote_util:172] INFO - connecting to 10.111.151.102 with username : root password : couchbase ssh_key: [2016-11-04 12:39:32,132] - [remote_util:206] INFO - Connected to 10.111.151.102 [2016-11-04 12:39:32,583] - [remote_util:2383] INFO - running command.raw on 10.111.151.102: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index*.pindex | wc -l [2016-11-04 12:39:32,664] - [remote_util:2420] INFO - command executed successfully [2016-11-04 12:39:32,665] - [fts_base:1407] INFO - 0 [2016-11-04 12:39:32,666] - [fts_base:869] INFO - Validated: all index files for default_index deleted from disk [2016-11-04 12:39:32,666] - [fts_base:1499] INFO - removing nodes from cluster ... [2016-11-04 12:39:32,672] - [fts_base:1501] INFO - cleanup [ip:10.111.151.101 port:8091 ssh_username:root, ip:10.111.151.102 port:8091 ssh_username:root] [2016-11-04 12:39:32,682] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 10.111.151.101 [2016-11-04 12:39:32,682] - [bucket_helper:140] INFO - remove bucket default ... [2016-11-04 12:39:34,387] - [bucket_helper:154] INFO - deleted bucket : default from 10.111.151.101 [2016-11-04 12:39:34,387] - [bucket_helper:230] INFO - waiting for bucket deletion to complete.... [2016-11-04 12:39:34,392] - [rest_client:134] INFO - node 10.111.151.101 existing buckets : [] [2016-11-04 12:39:34,410] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes [2016-11-04 12:39:34,413] - [rest_client:1300] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.111.151.102&user=Administrator&knownNodes=ns_1%4010.111.151.101%2Cns_1%4010.111.151.102 [2016-11-04 12:39:34,421] - [rest_client:1304] INFO - rebalance operation started [2016-11-04 12:39:34,426] - [rest_client:1435] INFO - rebalance percentage : 0.00 % [2016-11-04 12:39:54,440] - [rest_client:1390] INFO - rebalance progress took 20.0179619789 seconds [2016-11-04 12:39:54,440] - [rest_client:1391] INFO - sleep for 10 seconds after rebalance... [2016-11-04 12:40:04,451] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.111.151.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.151.102', 8091)] [2016-11-04 12:40:04,455] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.101:8091 [2016-11-04 12:40:04,459] - [cluster_helper:80] INFO - ns_server @ 10.111.151.101:8091 is running [2016-11-04 12:40:04,465] - [bucket_helper:138] INFO - deleting existing buckets [] on 10.111.151.102 [2016-11-04 12:40:04,477] - [cluster_helper:78] INFO - waiting for ns_server @ 10.111.151.102:8091 [2016-11-04 12:40:04,479] - [cluster_helper:80] INFO - ns_server @ 10.111.151.102:8091 is running Cluster instance shutdown with force [2016-11-04 12:40:04,480] - [fts_base:2606] INFO - ==== FTSbasetests cleanup is finished for test #6 test_sorting_of_results_on_non_indexed_fields === Cluster instance shutdown with force summary so far suite fts.stable_topology_fts.StableTopFTS , pass 1 , fail 5 failures so far... fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields testrunner logs, diags and results are available under /Users/mihirkamdar/PycharmProjects/testrunner_fts_1103/testrunner/logs/testrunner-16-Nov-04_12-33-27/test_6 Run after suite setup for fts.stable_topology_fts.StableTopFTS.test_sorting_of_results_on_non_indexed_fields Thread Cluster_Thread was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now.