./testrunner -i myfile.ini.bk get-cbcollect-info=True,get-logs=True,skip_cleanup=true -t view.createdeleteview.CreateDeleteViewTests.ddoc_ops_during_failover,ddoc_ops=delete,test_with_view=True,num_ddocs=2 ,num_views_per_ddoc=3,items=200000,nodes_out=2,replicas=2,sasl_buckets=2,sasl_bucket_priority=low:high Test Input params: {'skip_cleanup': 'true', 'ddoc_ops': 'delete', 'num_ddocs': '2', 'replicas': '2', 'case_number': 1, 'items': '200000', 'test_with_view': 'True', 'num_views_per_ddoc': '3', 'num_nodes': 4, 'cluster_name': ' myfile.ini', 'sasl_bucket_priority': 'low:high', 'nodes_out': '2', 'get-logs': 'True', 'sasl_buckets': '2', 'get-cbcollect-info': 'True', 'logs_folder': '/root/testrunner/logs/testrunner-14-Sep-01_12-35-39 /test_1', 'ini': 'myfile.ini.bk'} Run before suite setup for view.createdeleteview.CreateDeleteViewTests.ddoc_ops_during_failover 2014-09-01 12:35:39 | WARNING | MainProcess | MainThread | [data_helper.] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially ddoc_ops_during_failover (view.createdeleteview.CreateDeleteViewTests) ... 2014-09-01 12:35:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for t est #1 ddoc_ops_during_failover============== 2014-09-01 12:35:39 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force 2014-09-01 12:35:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.230:8091:username=Administrator&password=password&port=8091 2014-09-01 12:35:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2134&username=Administrator&password=password 2014-09-01 12:35:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.231:8091:username=Administrator&password=password&port=8091 2014-09-01 12:35:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2134&username=Administrator&password=password 2014-09-01 12:35:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.232:8091:username=Administrator&password=password&port=8091 2014-09-01 12:35:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2134&username=Administrator&password=password 2014-09-01 12:35:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.245:8091:username=Administrator&password=password&port=8091 2014-09-01 12:35:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2134&username=Administrator&password=password 2014-09-01 12:35:42 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.230:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumbe r=3&ramQuotaMB=711&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=2&replicaIndex=1&saslPassword=None 2014-09-01 12:35:43 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.017795085907 seconds to create bucket default 2014-09-01 12:35:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.230 to accept set ops 2014-09-01 12:35:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 default 2014-09-01 12:35:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 default 2014-09-01 12:35:46 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 711 2014-09-01 12:35:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.230:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumbe r=3&ramQuotaMB=711&proxyPort=11211&authType=sasl&name=bucket0&flushEnabled=1&replicaNumber=2&replicaIndex=1&saslPassword=password 2014-09-01 12:35:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.0174968242645 seconds to create bucket bucket0 2014-09-01 12:35:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.230:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumbe r=8&ramQuotaMB=711&proxyPort=11211&authType=sasl&name=bucket1&flushEnabled=1&replicaNumber=2&replicaIndex=1&saslPassword=password 2014-09-01 12:35:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.0103268623352 seconds to create bucket bucket1 2014-09-01 12:35:47 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.105.230 to accept set ops 2014-09-01 12:35:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket0 2014-09-01 12:35:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket0 2014-09-01 12:35:56 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'bucket0' was created with per node RAM quota: 711 2014-09-01 12:35:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket1 in 172.23.105.230 to accept set ops 2014-09-01 12:35:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket1 2014-09-01 12:35:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket1 2014-09-01 12:35:57 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'bucket1' was created with per node RAM quota: 711 2014-09-01 12:35:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 ddoc_ops_during_failover ============== 2014-09-01 12:35:57 | INFO | MainProcess | test_thread | [createdeleteview.ddoc_ops_during_failover] create a cluster of all the available servers 2014-09-01 12:35:58 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.231:8091 to cluster 2014-09-01 12:35:58 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.231:8091 to this cluster @172.23.105.230:8091 2014-09-01 12:36:05 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.232:8091 to cluster 2014-09-01 12:36:05 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.232:8091 to this cluster @172.23.105.230:8091 2014-09-01 12:36:10 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.245:8091 to cluster 2014-09-01 12:36:10 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.245:8091 to this cluster @172.23.105.230:8091 2014-09-01 12:36:19 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.245%2Cns_1%40172.23.10 5.231%2Cns_1%40172.23.105.230%2Cns_1%40172.23.105.232 2014-09-01 12:36:19 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2014-09-01 12:36:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2014-09-01 12:36:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 6.13 % 2014-09-01 12:36:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 14.30 % 2014-09-01 12:36:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 22.23 % 2014-09-01 12:36:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 30.23 % 2014-09-01 12:37:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 33.33 % 2014-09-01 12:37:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 40.83 % 2014-09-01 12:37:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 49.00 % 2014-09-01 12:37:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 56.89 % 2014-09-01 12:37:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 64.27 % 2014-09-01 12:38:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 67.72 % 2014-09-01 12:38:10 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 75.73 % 2014-09-01 12:38:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 83.70 % 2014-09-01 12:38:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 92.02 % 2014-09-01 12:38:40 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 98.41 % 2014-09-01 12:38:50 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 150.291898012 sec 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [basetestcase._load_doc_data_all_buckets] create 200000 documents... 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 default 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.231:11210 default 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.232:11210 default 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.245:11210 default 2014-09-01 12:38:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket0 2014-09-01 12:38:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.231:11210 bucket0 2014-09-01 12:38:51 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0 2014-09-01 12:38:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.232:11210 bucket0 2014-09-01 12:38:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.245:11210 bucket0 2014-09-01 12:38:52 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.230:11210 bucket1 2014-09-01 12:38:52 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0 2014-09-01 12:38:52 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.231:11210 bucket1 2014-09-01 12:38:52 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.232:11210 bucket1 2014-09-01 12:38:53 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.245:11210 bucket1 2014-09-01 12:38:54 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0 2014-09-01 12:39:33 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 50000 with exp:0 2014-09-01 12:39:36 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 50000 with exp:0 2014-09-01 12:39:38 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 50000 with exp:0 2014-09-01 12:40:20 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 100000 with exp:0 2014-09-01 12:40:23 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 100000 with exp:0 2014-09-01 12:40:26 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 100000 with exp:0 2014-09-01 12:41:06 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 150000 with exp:0 2014-09-01 12:41:10 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 150000 with exp:0 2014-09-01 12:41:12 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 150000 with exp:0 2014-09-01 12:41:53 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 200000 with exp:0 2014-09-01 12:41:54 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 200000 with exp:0 2014-09-01 12:41:55 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 200000 with exp:0 2014-09-01 12:41:55 | INFO | MainProcess | test_thread | [createdeleteview._execute_ddoc_ops] Processing Create DDoc Operation On Bucket default 2014-09-01 12:41:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//default/_design/dev_test0 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v10?stale=ok 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test0 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v11?stale=ok 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test0 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v12?stale=ok 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test0 2014-09-01 12:41:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//default/_design/dev_test1 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:41:56 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test1 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v11?stale=ok 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test1 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v12?stale=ok 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test1 2014-09-01 12:41:57 | INFO | MainProcess | test_thread | [createdeleteview._execute_ddoc_ops] Processing Create DDoc Operation On Bucket bucket0 2014-09-01 12:41:57 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//bucket0/_design/dev_test0 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test0/_view/v10?stale=ok 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test0 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test0/_view/v11?stale=ok 2014-09-01 12:41:57 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test0 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test0/_view/v12?stale=ok 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test0 2014-09-01 12:41:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//bucket0/_design/dev_test1 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test1 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v11?stale=ok 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test1 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v12?stale=ok 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test1 2014-09-01 12:41:58 | INFO | MainProcess | test_thread | [createdeleteview._execute_ddoc_ops] Processing Create DDoc Operation On Bucket bucket1 2014-09-01 12:41:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//bucket1/_design/dev_test0 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:58 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v10?stale=ok 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test0 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v11?stale=ok 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test0 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v12?stale=ok 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test0 2014-09-01 12:41:59 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092//bucket1/_design/dev_test1 error 404 reason: not_found {"error":"not_found","reason":"mis sing"} 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was created successfully in ddoc: dev_test1 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v11?stale=ok 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [task.check] view : v11 was created successfully in ddoc: dev_test1 2014-09-01 12:41:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v12?stale=ok 2014-09-01 12:42:00 | INFO | MainProcess | Cluster_Thread | [task.check] view : v12 was created successfully in ddoc: dev_test1 2014-09-01 12:42:00 | INFO | MainProcess | test_thread | [createdeleteview._async_execute_ddoc_ops] Processing Delete DDoc Operation On Bucket default 2014-09-01 12:42:00 | INFO | MainProcess | test_thread | [createdeleteview._async_execute_ddoc_ops] Processing Delete DDoc Operation On Bucket bucket0 2014-09-01 12:42:00 | INFO | MainProcess | test_thread | [createdeleteview._async_execute_ddoc_ops] Processing Delete DDoc Operation On Bucket bucket1 2014-09-01 12:42:00 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.105.232:8091 2014-09-01 12:42:01 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.105.232 successful 2014-09-01 12:42:01 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.105.245:8091 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.105.245 successful 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [task.execute] 0 seconds sleep after failover, for nodes to go pending.... 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:42:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092/default/_design/dev_test1/_view/v10?stale=ok error 404 reason: not_found {"error":"not_fo und","reason":"Error opening view `v10`, from set `default`, design document `_design/dev_test1`: {not_found,\n missing_na med_view}"} 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was successfully deleted in ddoc: dev_test1 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:42:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v10?stale=ok error 404 reason: not_found {"error":"not_fo und","reason":"Error opening view `v10`, from set `bucket0`, design document `_design/dev_test1`: {not_found,\n missing_na med_view}"} 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was successfully deleted in ddoc: dev_test1 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v10?stale=ok 2014-09-01 12:42:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v10?stale=ok error 404 reason: not_found {"error":"not_fo und","reason":"Error opening view `v10`, from set `bucket1`, design document `_design/dev_test1`: {not_found,\n missing_na med_view}"} 2014-09-01 12:42:03 | INFO | MainProcess | Cluster_Thread | [task.check] view : v10 was successfully deleted in ddoc: dev_test1 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] DDoc Validation Started 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test1 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test0 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v10": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test1 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test0 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v10": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test1 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] Database Document dev_test0 details : {"views": {"v12": {"map": "function (doc) { emit(doc.age, doc. first_name);}"}, "v10": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "v11": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_ops_all_buckets] DDoc Validation Successful 2014-09-01 12:42:03 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_data_all_buckets] DDoc Data Validation Started on bucket bucket1. Expected Data Items 200000 2014-09-01 12:42:04 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:42:50 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:43:00 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:43:00 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:43:01 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:43:11 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test1/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:43:21 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:43:21 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:43:22 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v10?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:43:39 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v10?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:43:49 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:43:49 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:43:50 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:10 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:44:10 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:44:11 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:21 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket1/_design/dev_test0/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:31 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:44:31 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:44:31 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_data_all_buckets] DDoc Data Validation Started on bucket default. Expected Data Items 200000 2014-09-01 12:44:32 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:49 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:44:59 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:44:59 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:45:00 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:45:10 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test1/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:45:20 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:45:20 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:45:21 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v10?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:45:38 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v10?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:45:48 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:45:48 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:45:49 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:45:59 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:46:09 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:46:09 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:46:10 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:46:20 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/default/_design/dev_test0/_view/v12?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:46:30 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (200000 rows) returned 2014-09-01 12:46:30 | INFO | MainProcess | Cluster_Thread | [task.check] expected number of rows: '200000' was found for view query 2014-09-01 12:46:31 | INFO | MainProcess | test_thread | [createdeleteview._verify_ddoc_data_all_buckets] DDoc Data Validation Started on bucket bucket0. Expected Data Items 200000 2014-09-01 12:46:31 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:46:49 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.230:8092/bucket0/_design/dev_test1/_view/v11?stale=false&connection_timeout=60000&full_se t=true 2014-09-01 12:46:59 | INFO | MainProcess | Cluster_Thread | [task.check] (200000 rows) expected, (199996 rows) returned FAIL 2014-09-01 12:46:59 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED ====================================================================== FAIL: ddoc_ops_during_failover (view.createdeleteview.CreateDeleteViewTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/view/createdeleteview.py", line 664, in ddoc_ops_during_failover self._verify_ddoc_data_all_buckets() File "pytests/view/createdeleteview.py", line 276, in _verify_ddoc_data_all_buckets self.fail("DDoc Data Validation Error: View - {0} in Design Doc - {1} and Bucket - {2}".format(view.name, ddoc_name, bucket)) AssertionError: DDoc Data Validation Error: View - v11 in Design Doc - dev_test1 and Bucket - bucket0