./testrunner -i /tmp/observe-tests.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=ALL -t observe.observetest.ObserveTests.test_observe_basic_data_load_delete,items=100,sasl_buckets=1,standard_buckets=1,rebalance=out,GROUP=P1

Test Input params:
{'logs_folder': '/jenkins/workspace/debian_x64--90_04--observe_tests-P0/logs/testrunner-14-Aug-21_01-01-43/test_30', 'GROUP': 'ALL', 'items': '100', 'standard_buckets': '1', 'conf_file': 'conf/py-observe.conf', 'num_nodes': 3, 'cluster_name': 'observe-tests', 'stop-on-failure': 'False', 'ini': '/tmp/observe-tests.ini', 'case_number': 30, 'get-logs': 'False', 'sasl_buckets': '1', 'get-cbcollect-info': 'True', 'rebalance': 'out', 'spec': 'py-observe'}
[2014-08-21 05:09:43,684] - [basetestcase:109] INFO - ==============  basetestcase setup was started for test #30 test_observe_basic_data_load_delete==============
[2014-08-21 05:09:44,695] - [rest_client:764] INFO - settings/web params on 10.3.5.154:8091:username=Administrator&password=password&port=8091
[2014-08-21 05:09:44,725] - [rest_client:784] INFO - pools/default params : memoryQuota=2064&username=Administrator&password=password
[2014-08-21 05:09:44,746] - [rest_client:764] INFO - settings/web params on 10.3.5.155:8091:username=Administrator&password=password&port=8091
[2014-08-21 05:09:44,767] - [rest_client:784] INFO - pools/default params : memoryQuota=2064&username=Administrator&password=password
[2014-08-21 05:09:44,782] - [rest_client:764] INFO - settings/web params on 10.3.2.146:8091:username=Administrator&password=password&port=8091
[2014-08-21 05:09:44,809] - [rest_client:784] INFO - pools/default params : memoryQuota=2064&username=Administrator&password=password
[2014-08-21 05:09:45,832] - [rest_client:1525] INFO - http://10.3.5.154:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=688&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2014-08-21 05:09:45,847] - [rest_client:1531] INFO - 0.0143859386444 seconds to create bucket default
[2014-08-21 05:09:45,848] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 10.3.5.154 to accept set ops
[2014-08-21 05:09:47,094] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:09:47,268] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:09:48,591] - [task:204] INFO - bucket 'default' was created with per node RAM quota: 688
[2014-08-21 05:09:49,616] - [rest_client:1525] INFO - http://10.3.5.154:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=688&proxyPort=11211&authType=sasl&name=bucket0&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=password
[2014-08-21 05:09:49,629] - [rest_client:1531] INFO - 0.0129389762878 seconds to create bucket bucket0
[2014-08-21 05:09:49,630] - [bucket_helper:305] INFO - waiting for memcached bucket : bucket0 in 10.3.5.154 to accept set ops
[2014-08-21 05:09:50,906] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:09:51,077] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:09:52,484] - [task:204] INFO - bucket 'bucket0' was created with per node RAM quota: 688
[2014-08-21 05:09:53,504] - [rest_client:1525] INFO - http://10.3.5.154:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=688
[2014-08-21 05:09:53,530] - [rest_client:1531] INFO - 0.0261259078979 seconds to create bucket standard_bucket0
[2014-08-21 05:09:53,531] - [bucket_helper:305] INFO - waiting for memcached bucket : standard_bucket0 in 10.3.5.154 to accept set ops
[2014-08-21 05:09:54,789] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:09:54,989] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:09:56,478] - [task:204] INFO - bucket 'standard_bucket0' was created with per node RAM quota: 688
[2014-08-21 05:09:56,486] - [basetestcase:170] INFO - ==============  basetestcase setup was finished for test #30 test_observe_basic_data_load_delete ==============
[2014-08-21 05:09:56,504] - [observetest:30] INFO - Observe Rebalance Started
[2014-08-21 05:09:57,480] - [task:300] INFO - adding node 10.3.5.155:8091 to cluster
[2014-08-21 05:09:57,481] - [rest_client:933] INFO - adding remote node @10.3.5.155:8091 to this cluster @10.3.5.154:8091
[2014-08-21 05:10:01,749] - [rest_client:1095] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.5.154%2Cns_1%4010.3.5.155
[2014-08-21 05:10:01,762] - [rest_client:1099] INFO - rebalance operation started
[2014-08-21 05:10:01,781] - [rest_client:1217] INFO - rebalance percentage : 0.00 %
[2014-08-21 05:10:11,806] - [rest_client:1217] INFO - rebalance percentage : 4.49 %
[2014-08-21 05:10:21,830] - [rest_client:1217] INFO - rebalance percentage : 12.24 %
[2014-08-21 05:10:32,008] - [rest_client:1217] INFO - rebalance percentage : 21.03 %
[2014-08-21 05:10:42,031] - [rest_client:1217] INFO - rebalance percentage : 27.24 %
[2014-08-21 05:10:52,058] - [rest_client:1217] INFO - rebalance percentage : 33.22 %
[2014-08-21 05:11:02,082] - [rest_client:1217] INFO - rebalance percentage : 39.71 %
[2014-08-21 05:11:12,105] - [rest_client:1217] INFO - rebalance percentage : 48.72 %
[2014-08-21 05:11:22,130] - [rest_client:1217] INFO - rebalance percentage : 57.05 %
[2014-08-21 05:11:32,152] - [rest_client:1217] INFO - rebalance percentage : 62.98 %
[2014-08-21 05:11:42,182] - [rest_client:1217] INFO - rebalance percentage : 68.58 %
[2014-08-21 05:11:52,201] - [rest_client:1217] INFO - rebalance percentage : 77.19 %
[2014-08-21 05:12:02,226] - [rest_client:1217] INFO - rebalance percentage : 86.52 %
[2014-08-21 05:12:12,250] - [rest_client:1217] INFO - rebalance percentage : 93.45 %
[2014-08-21 05:12:22,268] - [rest_client:1217] INFO - rebalance percentage : 99.89 %
[2014-08-21 05:12:32,287] - [task:411] INFO - rebalancing was completed with progress: 100% in 150.524060965 sec
[2014-08-21 05:12:32,380] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:12:32,481] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 default
[2014-08-21 05:12:32,633] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:12:32,746] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 bucket0
[2014-08-21 05:12:32,916] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:12:33,011] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 standard_bucket0
[2014-08-21 05:12:33,290] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-21 05:12:33,290] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-21 05:12:33,293] - [task:772] INFO - Batch create documents done #: 0 with exp:0
[2014-08-21 05:12:33,675] - [task:772] INFO - Batch create documents done #: 100 with exp:0
[2014-08-21 05:12:33,696] - [task:772] INFO - Batch create documents done #: 100 with exp:0
[2014-08-21 05:12:33,745] - [task:772] INFO - Batch create documents done #: 100 with exp:0
[2014-08-21 05:12:33,747] - [rebalance_helper:22] INFO - waiting for bucket default stat : ep_queue_size to match 0 on 10.3.5.154
[2014-08-21 05:12:33,808] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:33,809] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:12:33,950] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:33,950] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 default
[2014-08-21 05:12:34,031] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:34,031] - [rebalance_helper:51] INFO - ep_queue_size : 0
[2014-08-21 05:12:34,032] - [rebalance_helper:22] INFO - waiting for bucket default stat : ep_flusher_todo to match 0 on 10.3.5.154
[2014-08-21 05:12:34,094] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,094] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:12:34,209] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,210] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 default
[2014-08-21 05:12:34,293] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:34,294] - [rebalance_helper:51] INFO - ep_flusher_todo : 0
[2014-08-21 05:12:34,294] - [rebalance_helper:22] INFO - waiting for bucket default stat : ep_uncommitted_items to match 0 on 10.3.5.154
[2014-08-21 05:12:34,358] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,359] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:12:34,490] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,491] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 default
[2014-08-21 05:12:34,585] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:34,586] - [rebalance_helper:51] INFO - ep_uncommitted_items : 0
[2014-08-21 05:12:34,587] - [rebalance_helper:22] INFO - waiting for bucket bucket0 stat : ep_queue_size to match 0 on 10.3.5.154
[2014-08-21 05:12:34,658] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,658] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:12:34,783] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,784] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 bucket0
[2014-08-21 05:12:34,866] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:34,867] - [rebalance_helper:51] INFO - ep_queue_size : 0
[2014-08-21 05:12:34,868] - [rebalance_helper:22] INFO - waiting for bucket bucket0 stat : ep_flusher_todo to match 0 on 10.3.5.154
[2014-08-21 05:12:34,936] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:34,936] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:12:35,060] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,060] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 bucket0
[2014-08-21 05:12:35,141] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:35,141] - [rebalance_helper:51] INFO - ep_flusher_todo : 0
[2014-08-21 05:12:35,142] - [rebalance_helper:22] INFO - waiting for bucket bucket0 stat : ep_uncommitted_items to match 0 on 10.3.5.154
[2014-08-21 05:12:35,198] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,199] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:12:35,319] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,319] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 bucket0
[2014-08-21 05:12:35,407] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:35,408] - [rebalance_helper:51] INFO - ep_uncommitted_items : 0
[2014-08-21 05:12:35,408] - [rebalance_helper:22] INFO - waiting for bucket standard_bucket0 stat : ep_queue_size to match 0 on 10.3.5.154
[2014-08-21 05:12:35,473] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,473] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:12:35,600] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,601] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 standard_bucket0
[2014-08-21 05:12:35,695] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:35,695] - [rebalance_helper:51] INFO - ep_queue_size : 0
[2014-08-21 05:12:35,696] - [rebalance_helper:22] INFO - waiting for bucket standard_bucket0 stat : ep_flusher_todo to match 0 on 10.3.5.154
[2014-08-21 05:12:35,755] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,755] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:12:35,890] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:35,890] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 standard_bucket0
[2014-08-21 05:12:35,979] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:35,979] - [rebalance_helper:51] INFO - ep_flusher_todo : 0
[2014-08-21 05:12:35,980] - [rebalance_helper:22] INFO - waiting for bucket standard_bucket0 stat : ep_uncommitted_items to match 0 on 10.3.5.154
[2014-08-21 05:12:36,045] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.154', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:36,046] - [data_helper:293] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:12:36,175] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'10.3.5.155', 'password': 'password', 'port': 8091}
[2014-08-21 05:12:36,176] - [data_helper:293] INFO - creating direct client 10.3.5.155:11210 standard_bucket0
[2014-08-21 05:12:36,257] - [rebalance_helper:46] INFO - 0
[2014-08-21 05:12:36,258] - [rebalance_helper:51] INFO - ep_uncommitted_items : 0
[2014-08-21 05:12:36,335] - [rest_client:1095] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.5.155&user=Administrator&knownNodes=ns_1%4010.3.5.154%2Cns_1%4010.3.5.155
[2014-08-21 05:12:36,344] - [rest_client:1099] INFO - rebalance operation started
[2014-08-21 05:12:36,353] - [rest_client:747] ERROR - http://10.3.5.154:8092//default/_design/dev_Doc1 error 404 reason: not_found {"error":"not_found","reason":"missing"}
[2014-08-21 05:12:36,401] - [rest_client:1217] INFO - rebalance percentage : 0.00 %
[2014-08-21 05:12:36,402] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/default/_design/dev_Doc1/_view/default_view?stale=ok
[2014-08-21 05:12:41,441] - [task:1325] INFO - view : default_view was created successfully in ddoc: dev_Doc1
[2014-08-21 05:12:41,610] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 default
[2014-08-21 05:12:41,790] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 default
[2014-08-21 05:12:41,928] - [observetest:156] INFO - ##########key:-observe0################
[2014-08-21 05:12:41,928] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,929] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,929] - [observetest:159] INFO - Time2:- 0:00:00.007345
[2014-08-21 05:12:41,930] - [observetest:162] INFO - Max Time taken for observe is :- 0:00:00.007345
[2014-08-21 05:12:41,930] - [observetest:163] INFO - Cas Value:- 0
[2014-08-21 05:12:41,932] - [observetest:156] INFO - ##########key:-observe1################
[2014-08-21 05:12:41,932] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,933] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,933] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:41,942] - [observetest:156] INFO - ##########key:-observe2################
[2014-08-21 05:12:41,942] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,942] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,943] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:41,945] - [observetest:156] INFO - ##########key:-observe3################
[2014-08-21 05:12:41,946] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,946] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,947] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:41,951] - [observetest:156] INFO - ##########key:-observe4################
[2014-08-21 05:12:41,951] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,951] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,952] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:41,954] - [observetest:156] INFO - ##########key:-observe5################
[2014-08-21 05:12:41,954] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,955] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,955] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:41,960] - [observetest:156] INFO - ##########key:-observe6################
[2014-08-21 05:12:41,960] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,960] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,961] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:41,968] - [observetest:156] INFO - ##########key:-observe7################
[2014-08-21 05:12:41,969] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,969] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,970] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:41,974] - [observetest:156] INFO - ##########key:-observe8################
[2014-08-21 05:12:41,975] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,975] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,975] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:41,978] - [observetest:156] INFO - ##########key:-observe9################
[2014-08-21 05:12:41,978] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,978] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,979] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:41,980] - [observetest:156] INFO - ##########key:-observe10################
[2014-08-21 05:12:41,980] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,980] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,981] - [observetest:159] INFO - Time2:- 0:00:00.000021
[2014-08-21 05:12:41,983] - [observetest:156] INFO - ##########key:-observe11################
[2014-08-21 05:12:41,984] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,984] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,984] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:41,986] - [observetest:156] INFO - ##########key:-observe12################
[2014-08-21 05:12:41,986] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,987] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,987] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:41,988] - [observetest:156] INFO - ##########key:-observe13################
[2014-08-21 05:12:41,989] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,989] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,990] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:41,994] - [observetest:156] INFO - ##########key:-observe14################
[2014-08-21 05:12:41,994] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:41,995] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:41,996] - [observetest:159] INFO - Time2:- 0:00:00.000025
[2014-08-21 05:12:42,000] - [observetest:156] INFO - ##########key:-observe15################
[2014-08-21 05:12:42,000] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,001] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,001] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,003] - [observetest:156] INFO - ##########key:-observe16################
[2014-08-21 05:12:42,004] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,004] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,005] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,007] - [observetest:156] INFO - ##########key:-observe17################
[2014-08-21 05:12:42,008] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,008] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,009] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,011] - [observetest:156] INFO - ##########key:-observe18################
[2014-08-21 05:12:42,012] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,012] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,013] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:42,016] - [observetest:156] INFO - ##########key:-observe19################
[2014-08-21 05:12:42,017] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,017] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,018] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:42,020] - [observetest:156] INFO - ##########key:-observe20################
[2014-08-21 05:12:42,021] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,021] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,022] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,025] - [observetest:156] INFO - ##########key:-observe21################
[2014-08-21 05:12:42,026] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,026] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,026] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:42,032] - [observetest:156] INFO - ##########key:-observe22################
[2014-08-21 05:12:42,032] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,032] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,033] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:42,036] - [observetest:156] INFO - ##########key:-observe23################
[2014-08-21 05:12:42,036] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,036] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,036] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,041] - [observetest:156] INFO - ##########key:-observe24################
[2014-08-21 05:12:42,041] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,041] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,042] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:42,043] - [observetest:156] INFO - ##########key:-observe25################
[2014-08-21 05:12:42,043] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,044] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,044] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:42,046] - [observetest:156] INFO - ##########key:-observe26################
[2014-08-21 05:12:42,046] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,047] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,047] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:42,055] - [observetest:156] INFO - ##########key:-observe27################
[2014-08-21 05:12:42,055] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,055] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,055] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:42,057] - [observetest:156] INFO - ##########key:-observe28################
[2014-08-21 05:12:42,057] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,058] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,058] - [observetest:159] INFO - Time2:- 0:00:00.000020
[2014-08-21 05:12:42,062] - [observetest:156] INFO - ##########key:-observe29################
[2014-08-21 05:12:42,062] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,062] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,063] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:42,064] - [observetest:156] INFO - ##########key:-observe30################
[2014-08-21 05:12:42,064] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,064] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,065] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:42,068] - [observetest:156] INFO - ##########key:-observe31################
[2014-08-21 05:12:42,068] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,069] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,069] - [observetest:159] INFO - Time2:- 0:00:00.000021
[2014-08-21 05:12:42,073] - [observetest:156] INFO - ##########key:-observe32################
[2014-08-21 05:12:42,073] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,073] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,074] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,075] - [observetest:156] INFO - ##########key:-observe33################
[2014-08-21 05:12:42,075] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,075] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,075] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,081] - [observetest:156] INFO - ##########key:-observe34################
[2014-08-21 05:12:42,081] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,082] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,082] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,083] - [observetest:156] INFO - ##########key:-observe35################
[2014-08-21 05:12:42,084] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,084] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,085] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,086] - [observetest:156] INFO - ##########key:-observe36################
[2014-08-21 05:12:42,087] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,088] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,088] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,090] - [observetest:156] INFO - ##########key:-observe37################
[2014-08-21 05:12:42,090] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,091] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,091] - [observetest:159] INFO - Time2:- 0:00:00.000044
[2014-08-21 05:12:42,093] - [observetest:156] INFO - ##########key:-observe38################
[2014-08-21 05:12:42,094] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,094] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,095] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,099] - [observetest:156] INFO - ##########key:-observe39################
[2014-08-21 05:12:42,100] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,100] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,101] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,103] - [observetest:156] INFO - ##########key:-observe40################
[2014-08-21 05:12:42,104] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,104] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,105] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,106] - [observetest:156] INFO - ##########key:-observe41################
[2014-08-21 05:12:42,107] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,108] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,109] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,110] - [observetest:156] INFO - ##########key:-observe42################
[2014-08-21 05:12:42,111] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,111] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,112] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,114] - [observetest:156] INFO - ##########key:-observe43################
[2014-08-21 05:12:42,115] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,115] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,116] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,117] - [observetest:156] INFO - ##########key:-observe44################
[2014-08-21 05:12:42,118] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,119] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,120] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,147] - [observetest:156] INFO - ##########key:-observe45################
[2014-08-21 05:12:42,147] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,148] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,148] - [observetest:159] INFO - Time2:- 0:00:00.000019
[2014-08-21 05:12:42,149] - [observetest:156] INFO - ##########key:-observe46################
[2014-08-21 05:12:42,150] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,151] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,152] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,153] - [observetest:156] INFO - ##########key:-observe47################
[2014-08-21 05:12:42,154] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,155] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,155] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,157] - [observetest:156] INFO - ##########key:-observe48################
[2014-08-21 05:12:42,157] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,157] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,158] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,161] - [observetest:156] INFO - ##########key:-observe49################
[2014-08-21 05:12:42,161] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,162] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,163] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,165] - [observetest:156] INFO - ##########key:-observe50################
[2014-08-21 05:12:42,165] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,166] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,166] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,168] - [observetest:156] INFO - ##########key:-observe51################
[2014-08-21 05:12:42,168] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,169] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,169] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,171] - [observetest:156] INFO - ##########key:-observe52################
[2014-08-21 05:12:42,172] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,173] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,173] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,179] - [observetest:156] INFO - ##########key:-observe53################
[2014-08-21 05:12:42,179] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,181] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,182] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:42,184] - [observetest:156] INFO - ##########key:-observe54################
[2014-08-21 05:12:42,185] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,186] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,187] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:42,190] - [observetest:156] INFO - ##########key:-observe55################
[2014-08-21 05:12:42,191] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,191] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,192] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:42,198] - [observetest:156] INFO - ##########key:-observe56################
[2014-08-21 05:12:42,199] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,199] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,200] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,201] - [observetest:156] INFO - ##########key:-observe57################
[2014-08-21 05:12:42,202] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,203] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,203] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,205] - [observetest:156] INFO - ##########key:-observe58################
[2014-08-21 05:12:42,205] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,206] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,206] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,209] - [observetest:156] INFO - ##########key:-observe59################
[2014-08-21 05:12:42,210] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,210] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,211] - [observetest:159] INFO - Time2:- 0:00:00.000019
[2014-08-21 05:12:42,213] - [observetest:156] INFO - ##########key:-observe60################
[2014-08-21 05:12:42,213] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,214] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,215] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,217] - [observetest:156] INFO - ##########key:-observe61################
[2014-08-21 05:12:42,217] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,217] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,218] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,224] - [observetest:156] INFO - ##########key:-observe62################
[2014-08-21 05:12:42,225] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,225] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,226] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,227] - [observetest:156] INFO - ##########key:-observe63################
[2014-08-21 05:12:42,228] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,228] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,229] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,232] - [observetest:156] INFO - ##########key:-observe64################
[2014-08-21 05:12:42,233] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,233] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,234] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,235] - [observetest:156] INFO - ##########key:-observe65################
[2014-08-21 05:12:42,236] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,236] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,237] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,238] - [observetest:156] INFO - ##########key:-observe66################
[2014-08-21 05:12:42,239] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,240] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,240] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,241] - [observetest:156] INFO - ##########key:-observe67################
[2014-08-21 05:12:42,242] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,242] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,243] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,244] - [observetest:156] INFO - ##########key:-observe68################
[2014-08-21 05:12:42,245] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,245] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,246] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,247] - [observetest:156] INFO - ##########key:-observe69################
[2014-08-21 05:12:42,248] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,248] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,249] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,250] - [observetest:156] INFO - ##########key:-observe70################
[2014-08-21 05:12:42,251] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,252] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,252] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,255] - [observetest:156] INFO - ##########key:-observe71################
[2014-08-21 05:12:42,256] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,256] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,257] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,259] - [observetest:156] INFO - ##########key:-observe72################
[2014-08-21 05:12:42,259] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,260] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,260] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,262] - [observetest:156] INFO - ##########key:-observe73################
[2014-08-21 05:12:42,262] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,263] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,263] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,265] - [observetest:156] INFO - ##########key:-observe74################
[2014-08-21 05:12:42,265] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,266] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,267] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,268] - [observetest:156] INFO - ##########key:-observe75################
[2014-08-21 05:12:42,268] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,269] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,269] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,271] - [observetest:156] INFO - ##########key:-observe76################
[2014-08-21 05:12:42,272] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,272] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,273] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,274] - [observetest:156] INFO - ##########key:-observe77################
[2014-08-21 05:12:42,275] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,276] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,276] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,278] - [observetest:156] INFO - ##########key:-observe78################
[2014-08-21 05:12:42,279] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,279] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,280] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,282] - [observetest:156] INFO - ##########key:-observe79################
[2014-08-21 05:12:42,283] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,283] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,284] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,288] - [observetest:156] INFO - ##########key:-observe80################
[2014-08-21 05:12:42,288] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,289] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,290] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,291] - [observetest:156] INFO - ##########key:-observe81################
[2014-08-21 05:12:42,292] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,292] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,293] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,295] - [observetest:156] INFO - ##########key:-observe82################
[2014-08-21 05:12:42,295] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,296] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,297] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,301] - [observetest:156] INFO - ##########key:-observe83################
[2014-08-21 05:12:42,302] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,302] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,303] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,304] - [observetest:156] INFO - ##########key:-observe84################
[2014-08-21 05:12:42,305] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,306] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,306] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,309] - [observetest:156] INFO - ##########key:-observe85################
[2014-08-21 05:12:42,309] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,310] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,310] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,314] - [observetest:156] INFO - ##########key:-observe86################
[2014-08-21 05:12:42,315] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,316] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,316] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,318] - [observetest:156] INFO - ##########key:-observe87################
[2014-08-21 05:12:42,318] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,319] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,319] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,321] - [observetest:156] INFO - ##########key:-observe88################
[2014-08-21 05:12:42,322] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,322] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,323] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,327] - [observetest:156] INFO - ##########key:-observe89################
[2014-08-21 05:12:42,327] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,328] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,328] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,330] - [observetest:156] INFO - ##########key:-observe90################
[2014-08-21 05:12:42,330] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,331] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,331] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,334] - [observetest:156] INFO - ##########key:-observe91################
[2014-08-21 05:12:42,334] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,335] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,336] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,337] - [observetest:156] INFO - ##########key:-observe92################
[2014-08-21 05:12:42,338] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,338] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,339] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,342] - [observetest:156] INFO - ##########key:-observe93################
[2014-08-21 05:12:42,343] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,343] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,344] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:42,348] - [observetest:156] INFO - ##########key:-observe94################
[2014-08-21 05:12:42,349] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,350] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,350] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,354] - [observetest:156] INFO - ##########key:-observe95################
[2014-08-21 05:12:42,355] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,355] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,355] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,360] - [observetest:156] INFO - ##########key:-observe96################
[2014-08-21 05:12:42,360] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,360] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,361] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:42,363] - [observetest:156] INFO - ##########key:-observe97################
[2014-08-21 05:12:42,364] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,365] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,365] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,367] - [observetest:156] INFO - ##########key:-observe98################
[2014-08-21 05:12:42,368] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,369] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,369] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:42,371] - [observetest:156] INFO - ##########key:-observe99################
[2014-08-21 05:12:42,372] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:42,372] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:42,373] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:42,470] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/default/_design/dev_Doc1/_view/default_view?stale=false&connection_timeout=60000&full_set=true
[2014-08-21 05:12:44,183] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/default/_design/dev_Doc1/_view/default_view?stale=false&connection_timeout=60000&full_set=true
[2014-08-21 05:12:44,468] - [task:1535] INFO - (100 rows) expected, (100 rows) returned
[2014-08-21 05:12:44,469] - [task:1542] INFO - expected number of rows: '100' was found for view query
[2014-08-21 05:12:44,495] - [observetest:166] INFO - Observe Validation:- view: {"default_view": {"map": "function (doc) { emit(null, doc);}"}} in design doc dev_Doc1 and in bucket default
[2014-08-21 05:12:45,487] - [rest_client:747] ERROR - http://10.3.5.154:8092//bucket0/_design/dev_Doc1 error 404 reason: not_found {"error":"not_found","reason":"missing"}
[2014-08-21 05:12:45,639] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/bucket0/_design/dev_Doc1/_view/default_view?stale=ok
[2014-08-21 05:12:45,682] - [task:1325] INFO - view : default_view was created successfully in ddoc: dev_Doc1
[2014-08-21 05:12:45,956] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 bucket0
[2014-08-21 05:12:46,101] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 bucket0
[2014-08-21 05:12:46,224] - [observetest:156] INFO - ##########key:-observe0################
[2014-08-21 05:12:46,224] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,225] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,225] - [observetest:159] INFO - Time2:- 0:00:00.000025
[2014-08-21 05:12:46,226] - [observetest:162] INFO - Max Time taken for observe is :- 0:00:00.000025
[2014-08-21 05:12:46,227] - [observetest:163] INFO - Cas Value:- 0
[2014-08-21 05:12:46,229] - [observetest:156] INFO - ##########key:-observe1################
[2014-08-21 05:12:46,230] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,230] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,231] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,233] - [observetest:156] INFO - ##########key:-observe2################
[2014-08-21 05:12:46,234] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,234] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,235] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,237] - [observetest:156] INFO - ##########key:-observe3################
[2014-08-21 05:12:46,238] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,238] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,239] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:46,244] - [observetest:156] INFO - ##########key:-observe4################
[2014-08-21 05:12:46,245] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,245] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,246] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:46,248] - [observetest:156] INFO - ##########key:-observe5################
[2014-08-21 05:12:46,249] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,249] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,250] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,252] - [observetest:156] INFO - ##########key:-observe6################
[2014-08-21 05:12:46,253] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,253] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,254] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,258] - [observetest:156] INFO - ##########key:-observe7################
[2014-08-21 05:12:46,258] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,259] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,259] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,263] - [observetest:156] INFO - ##########key:-observe8################
[2014-08-21 05:12:46,264] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,264] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,265] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,267] - [observetest:156] INFO - ##########key:-observe9################
[2014-08-21 05:12:46,268] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,268] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,269] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,271] - [observetest:156] INFO - ##########key:-observe10################
[2014-08-21 05:12:46,272] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,272] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,273] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,274] - [observetest:156] INFO - ##########key:-observe11################
[2014-08-21 05:12:46,275] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,275] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,276] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,278] - [observetest:156] INFO - ##########key:-observe12################
[2014-08-21 05:12:46,278] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,279] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,280] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,281] - [observetest:156] INFO - ##########key:-observe13################
[2014-08-21 05:12:46,281] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,282] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,282] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,283] - [observetest:156] INFO - ##########key:-observe14################
[2014-08-21 05:12:46,284] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,284] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,285] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,286] - [observetest:156] INFO - ##########key:-observe15################
[2014-08-21 05:12:46,287] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,287] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,288] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,293] - [observetest:156] INFO - ##########key:-observe16################
[2014-08-21 05:12:46,294] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,295] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,295] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,298] - [observetest:156] INFO - ##########key:-observe17################
[2014-08-21 05:12:46,298] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,299] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,300] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,302] - [observetest:156] INFO - ##########key:-observe18################
[2014-08-21 05:12:46,303] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,304] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,304] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,307] - [observetest:156] INFO - ##########key:-observe19################
[2014-08-21 05:12:46,307] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,307] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,308] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,310] - [observetest:156] INFO - ##########key:-observe20################
[2014-08-21 05:12:46,311] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,312] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,312] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,314] - [observetest:156] INFO - ##########key:-observe21################
[2014-08-21 05:12:46,315] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,315] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,316] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,318] - [observetest:156] INFO - ##########key:-observe22################
[2014-08-21 05:12:46,319] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,319] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,320] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,322] - [observetest:156] INFO - ##########key:-observe23################
[2014-08-21 05:12:46,323] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,324] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,324] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,325] - [observetest:156] INFO - ##########key:-observe24################
[2014-08-21 05:12:46,326] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,326] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,327] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,329] - [observetest:156] INFO - ##########key:-observe25################
[2014-08-21 05:12:46,330] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,330] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,331] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,334] - [observetest:156] INFO - ##########key:-observe26################
[2014-08-21 05:12:46,335] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,336] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,336] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,338] - [observetest:156] INFO - ##########key:-observe27################
[2014-08-21 05:12:46,338] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,339] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,340] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,342] - [observetest:156] INFO - ##########key:-observe28################
[2014-08-21 05:12:46,343] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,343] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,344] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,346] - [observetest:156] INFO - ##########key:-observe29################
[2014-08-21 05:12:46,347] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,347] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,348] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,350] - [observetest:156] INFO - ##########key:-observe30################
[2014-08-21 05:12:46,350] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,351] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,352] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,355] - [observetest:156] INFO - ##########key:-observe31################
[2014-08-21 05:12:46,356] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,356] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,357] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,359] - [observetest:156] INFO - ##########key:-observe32################
[2014-08-21 05:12:46,359] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,360] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,361] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,363] - [observetest:156] INFO - ##########key:-observe33################
[2014-08-21 05:12:46,363] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,364] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,365] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,367] - [observetest:156] INFO - ##########key:-observe34################
[2014-08-21 05:12:46,367] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,368] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,369] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,372] - [observetest:156] INFO - ##########key:-observe35################
[2014-08-21 05:12:46,373] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,373] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,374] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,376] - [observetest:156] INFO - ##########key:-observe36################
[2014-08-21 05:12:46,376] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,377] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,378] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,381] - [observetest:156] INFO - ##########key:-observe37################
[2014-08-21 05:12:46,382] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,382] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,383] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,385] - [observetest:156] INFO - ##########key:-observe38################
[2014-08-21 05:12:46,386] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,386] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,387] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,390] - [observetest:156] INFO - ##########key:-observe39################
[2014-08-21 05:12:46,391] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,391] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,392] - [observetest:159] INFO - Time2:- 0:00:00.000029
[2014-08-21 05:12:46,392] - [observetest:162] INFO - Max Time taken for observe is :- 0:00:00.000029
[2014-08-21 05:12:46,393] - [observetest:163] INFO - Cas Value:- 0
[2014-08-21 05:12:46,395] - [observetest:156] INFO - ##########key:-observe40################
[2014-08-21 05:12:46,395] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,396] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,397] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,399] - [observetest:156] INFO - ##########key:-observe41################
[2014-08-21 05:12:46,400] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,400] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,401] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,402] - [observetest:156] INFO - ##########key:-observe42################
[2014-08-21 05:12:46,403] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,403] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,404] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,407] - [observetest:156] INFO - ##########key:-observe43################
[2014-08-21 05:12:46,407] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,408] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,409] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,411] - [observetest:156] INFO - ##########key:-observe44################
[2014-08-21 05:12:46,412] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,412] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,413] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,414] - [observetest:156] INFO - ##########key:-observe45################
[2014-08-21 05:12:46,414] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,415] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,416] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,418] - [observetest:156] INFO - ##########key:-observe46################
[2014-08-21 05:12:46,419] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,419] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,420] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,424] - [observetest:156] INFO - ##########key:-observe47################
[2014-08-21 05:12:46,425] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,425] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,426] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,427] - [observetest:156] INFO - ##########key:-observe48################
[2014-08-21 05:12:46,427] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,428] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,429] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,430] - [observetest:156] INFO - ##########key:-observe49################
[2014-08-21 05:12:46,431] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,431] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,432] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,433] - [observetest:156] INFO - ##########key:-observe50################
[2014-08-21 05:12:46,434] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,435] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,435] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,441] - [observetest:156] INFO - ##########key:-observe51################
[2014-08-21 05:12:46,443] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,444] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,444] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,446] - [observetest:156] INFO - ##########key:-observe52################
[2014-08-21 05:12:46,446] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,447] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,447] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,456] - [observetest:156] INFO - ##########key:-observe53################
[2014-08-21 05:12:46,456] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,457] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,457] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,459] - [observetest:156] INFO - ##########key:-observe54################
[2014-08-21 05:12:46,460] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,460] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,461] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,464] - [observetest:156] INFO - ##########key:-observe55################
[2014-08-21 05:12:46,465] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,466] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,467] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,469] - [observetest:156] INFO - ##########key:-observe56################
[2014-08-21 05:12:46,470] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,471] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,471] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,475] - [observetest:156] INFO - ##########key:-observe57################
[2014-08-21 05:12:46,475] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,476] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,476] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,478] - [observetest:156] INFO - ##########key:-observe58################
[2014-08-21 05:12:46,479] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,480] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,480] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,483] - [observetest:156] INFO - ##########key:-observe59################
[2014-08-21 05:12:46,484] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,484] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,485] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,486] - [observetest:156] INFO - ##########key:-observe60################
[2014-08-21 05:12:46,487] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,488] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,488] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,490] - [observetest:156] INFO - ##########key:-observe61################
[2014-08-21 05:12:46,490] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,491] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,492] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,496] - [observetest:156] INFO - ##########key:-observe62################
[2014-08-21 05:12:46,497] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,497] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,498] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,500] - [observetest:156] INFO - ##########key:-observe63################
[2014-08-21 05:12:46,500] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,501] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,502] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:46,504] - [observetest:156] INFO - ##########key:-observe64################
[2014-08-21 05:12:46,504] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,505] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,505] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,508] - [observetest:156] INFO - ##########key:-observe65################
[2014-08-21 05:12:46,509] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,509] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,510] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,512] - [observetest:156] INFO - ##########key:-observe66################
[2014-08-21 05:12:46,513] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,514] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,514] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,516] - [observetest:156] INFO - ##########key:-observe67################
[2014-08-21 05:12:46,516] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,516] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,517] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,519] - [observetest:156] INFO - ##########key:-observe68################
[2014-08-21 05:12:46,520] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,521] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,521] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:46,527] - [observetest:156] INFO - ##########key:-observe69################
[2014-08-21 05:12:46,528] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,529] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,529] - [observetest:159] INFO - Time2:- 0:00:00.000026
[2014-08-21 05:12:46,532] - [observetest:156] INFO - ##########key:-observe70################
[2014-08-21 05:12:46,532] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,533] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,533] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:46,535] - [observetest:156] INFO - ##########key:-observe71################
[2014-08-21 05:12:46,535] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,536] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,537] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,538] - [observetest:156] INFO - ##########key:-observe72################
[2014-08-21 05:12:46,539] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,539] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,540] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,542] - [observetest:156] INFO - ##########key:-observe73################
[2014-08-21 05:12:46,542] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,543] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,543] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,547] - [observetest:156] INFO - ##########key:-observe74################
[2014-08-21 05:12:46,547] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,548] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,548] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,552] - [observetest:156] INFO - ##########key:-observe75################
[2014-08-21 05:12:46,553] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,553] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,553] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,555] - [observetest:156] INFO - ##########key:-observe76################
[2014-08-21 05:12:46,556] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,556] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,557] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,559] - [observetest:156] INFO - ##########key:-observe77################
[2014-08-21 05:12:46,560] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,560] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,561] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,566] - [observetest:156] INFO - ##########key:-observe78################
[2014-08-21 05:12:46,566] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,567] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,567] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,569] - [observetest:156] INFO - ##########key:-observe79################
[2014-08-21 05:12:46,569] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,570] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,571] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,572] - [observetest:156] INFO - ##########key:-observe80################
[2014-08-21 05:12:46,573] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,573] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,574] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,575] - [observetest:156] INFO - ##########key:-observe81################
[2014-08-21 05:12:46,576] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,577] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,577] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,579] - [observetest:156] INFO - ##########key:-observe82################
[2014-08-21 05:12:46,580] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,580] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,581] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:46,583] - [observetest:156] INFO - ##########key:-observe83################
[2014-08-21 05:12:46,583] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,584] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,585] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,586] - [observetest:156] INFO - ##########key:-observe84################
[2014-08-21 05:12:46,587] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,587] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,588] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,590] - [observetest:156] INFO - ##########key:-observe85################
[2014-08-21 05:12:46,590] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,591] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,591] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,593] - [observetest:156] INFO - ##########key:-observe86################
[2014-08-21 05:12:46,594] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,594] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,595] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:46,603] - [observetest:156] INFO - ##########key:-observe87################
[2014-08-21 05:12:46,604] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,604] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,604] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:46,606] - [observetest:156] INFO - ##########key:-observe88################
[2014-08-21 05:12:46,607] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,607] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,608] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,610] - [observetest:156] INFO - ##########key:-observe89################
[2014-08-21 05:12:46,610] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,611] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,612] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,614] - [observetest:156] INFO - ##########key:-observe90################
[2014-08-21 05:12:46,614] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,614] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,615] - [observetest:159] INFO - Time2:- 0:00:00.000026
[2014-08-21 05:12:46,620] - [observetest:156] INFO - ##########key:-observe91################
[2014-08-21 05:12:46,620] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,621] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,621] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:46,626] - [observetest:156] INFO - ##########key:-observe92################
[2014-08-21 05:12:46,627] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,628] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,628] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,630] - [observetest:156] INFO - ##########key:-observe93################
[2014-08-21 05:12:46,631] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,631] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,632] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,637] - [observetest:156] INFO - ##########key:-observe94################
[2014-08-21 05:12:46,638] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,638] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,638] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,640] - [observetest:156] INFO - ##########key:-observe95################
[2014-08-21 05:12:46,640] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,641] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,641] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,643] - [observetest:156] INFO - ##########key:-observe96################
[2014-08-21 05:12:46,643] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,644] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,645] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,647] - [observetest:156] INFO - ##########key:-observe97################
[2014-08-21 05:12:46,648] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,648] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,649] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,654] - [observetest:156] INFO - ##########key:-observe98################
[2014-08-21 05:12:46,655] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,656] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,657] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:46,658] - [observetest:156] INFO - ##########key:-observe99################
[2014-08-21 05:12:46,659] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:46,659] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:46,660] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:46,757] - [rest_client:1217] INFO - rebalance percentage : 7.64 %
[2014-08-21 05:12:46,774] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/bucket0/_design/dev_Doc1/_view/default_view?stale=false&connection_timeout=60000&full_set=true
[2014-08-21 05:12:48,545] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/bucket0/_design/dev_Doc1/_view/default_view?stale=false&connection_timeout=60000&full_set=true
[2014-08-21 05:12:48,811] - [task:1535] INFO - (100 rows) expected, (100 rows) returned
[2014-08-21 05:12:48,812] - [task:1542] INFO - expected number of rows: '100' was found for view query
[2014-08-21 05:12:48,831] - [observetest:166] INFO - Observe Validation:- view: {"default_view": {"map": "function (doc) { emit(null, doc);}"}} in design doc dev_Doc1 and in bucket bucket0
[2014-08-21 05:12:49,824] - [rest_client:747] ERROR - http://10.3.5.154:8092//standard_bucket0/_design/dev_Doc1 error 404 reason: not_found {"error":"not_found","reason":"missing"}
[2014-08-21 05:12:49,886] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/standard_bucket0/_design/dev_Doc1/_view/default_view?stale=ok
[2014-08-21 05:12:55,022] - [task:1325] INFO - view : default_view was created successfully in ddoc: dev_Doc1
[2014-08-21 05:12:55,248] - [data_helper:295] INFO - creating direct client 10.3.5.154:11210 standard_bucket0
[2014-08-21 05:12:55,441] - [data_helper:295] INFO - creating direct client 10.3.5.155:11210 standard_bucket0
[2014-08-21 05:12:55,579] - [observetest:156] INFO - ##########key:-observe0################
[2014-08-21 05:12:55,579] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,580] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,580] - [observetest:159] INFO - Time2:- 0:00:00.000027
[2014-08-21 05:12:55,580] - [observetest:162] INFO - Max Time taken for observe is :- 0:00:00.000027
[2014-08-21 05:12:55,581] - [observetest:163] INFO - Cas Value:- 0
[2014-08-21 05:12:55,584] - [observetest:156] INFO - ##########key:-observe1################
[2014-08-21 05:12:55,584] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,585] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,586] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:55,587] - [observetest:156] INFO - ##########key:-observe2################
[2014-08-21 05:12:55,588] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,588] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,589] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,590] - [observetest:156] INFO - ##########key:-observe3################
[2014-08-21 05:12:55,591] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,592] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,592] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,596] - [observetest:156] INFO - ##########key:-observe4################
[2014-08-21 05:12:55,597] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,598] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,598] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,601] - [observetest:156] INFO - ##########key:-observe5################
[2014-08-21 05:12:55,602] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,603] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,603] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,607] - [observetest:156] INFO - ##########key:-observe6################
[2014-08-21 05:12:55,607] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,608] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,608] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,610] - [observetest:156] INFO - ##########key:-observe7################
[2014-08-21 05:12:55,611] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,611] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,612] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,613] - [observetest:156] INFO - ##########key:-observe8################
[2014-08-21 05:12:55,614] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,614] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,615] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,617] - [observetest:156] INFO - ##########key:-observe9################
[2014-08-21 05:12:55,618] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,618] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,619] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,624] - [observetest:156] INFO - ##########key:-observe10################
[2014-08-21 05:12:55,624] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,625] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,625] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,628] - [observetest:156] INFO - ##########key:-observe11################
[2014-08-21 05:12:55,628] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,629] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,630] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,632] - [observetest:156] INFO - ##########key:-observe12################
[2014-08-21 05:12:55,633] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,633] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,634] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:55,640] - [observetest:156] INFO - ##########key:-observe13################
[2014-08-21 05:12:55,640] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,641] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,641] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,644] - [observetest:156] INFO - ##########key:-observe14################
[2014-08-21 05:12:55,644] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,645] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,646] - [observetest:159] INFO - Time2:- 0:00:00.000405
[2014-08-21 05:12:55,647] - [observetest:162] INFO - Max Time taken for observe is :- 0:00:00.000405
[2014-08-21 05:12:55,648] - [observetest:163] INFO - Cas Value:- 0
[2014-08-21 05:12:55,650] - [observetest:156] INFO - ##########key:-observe15################
[2014-08-21 05:12:55,650] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,651] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,651] - [observetest:159] INFO - Time2:- 0:00:00.000021
[2014-08-21 05:12:55,654] - [observetest:156] INFO - ##########key:-observe16################
[2014-08-21 05:12:55,655] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,656] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,656] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,662] - [observetest:156] INFO - ##########key:-observe17################
[2014-08-21 05:12:55,662] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,663] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,663] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,666] - [observetest:156] INFO - ##########key:-observe18################
[2014-08-21 05:12:55,667] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,667] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,668] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,671] - [observetest:156] INFO - ##########key:-observe19################
[2014-08-21 05:12:55,672] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,672] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,672] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,675] - [observetest:156] INFO - ##########key:-observe20################
[2014-08-21 05:12:55,675] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,676] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,677] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,680] - [observetest:156] INFO - ##########key:-observe21################
[2014-08-21 05:12:55,680] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,681] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,681] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,683] - [observetest:156] INFO - ##########key:-observe22################
[2014-08-21 05:12:55,683] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,684] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,685] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,686] - [observetest:156] INFO - ##########key:-observe23################
[2014-08-21 05:12:55,686] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,687] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,688] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,689] - [observetest:156] INFO - ##########key:-observe24################
[2014-08-21 05:12:55,690] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,691] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,691] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,692] - [observetest:156] INFO - ##########key:-observe25################
[2014-08-21 05:12:55,693] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,694] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,694] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,696] - [observetest:156] INFO - ##########key:-observe26################
[2014-08-21 05:12:55,697] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,697] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,698] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,700] - [observetest:156] INFO - ##########key:-observe27################
[2014-08-21 05:12:55,701] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,701] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,702] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,704] - [observetest:156] INFO - ##########key:-observe28################
[2014-08-21 05:12:55,705] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,706] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,706] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,709] - [observetest:156] INFO - ##########key:-observe29################
[2014-08-21 05:12:55,709] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,710] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,710] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,714] - [observetest:156] INFO - ##########key:-observe30################
[2014-08-21 05:12:55,714] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,714] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,715] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,716] - [observetest:156] INFO - ##########key:-observe31################
[2014-08-21 05:12:55,717] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,718] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,719] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,720] - [observetest:156] INFO - ##########key:-observe32################
[2014-08-21 05:12:55,721] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,722] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,723] - [observetest:159] INFO - Time2:- 0:00:00.000021
[2014-08-21 05:12:55,724] - [observetest:156] INFO - ##########key:-observe33################
[2014-08-21 05:12:55,725] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,725] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,726] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,727] - [observetest:156] INFO - ##########key:-observe34################
[2014-08-21 05:12:55,728] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,729] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,730] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,731] - [observetest:156] INFO - ##########key:-observe35################
[2014-08-21 05:12:55,731] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,732] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,733] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,735] - [observetest:156] INFO - ##########key:-observe36################
[2014-08-21 05:12:55,735] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,736] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,737] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,739] - [observetest:156] INFO - ##########key:-observe37################
[2014-08-21 05:12:55,739] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,740] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,740] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,741] - [observetest:156] INFO - ##########key:-observe38################
[2014-08-21 05:12:55,742] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,742] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,743] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,745] - [observetest:156] INFO - ##########key:-observe39################
[2014-08-21 05:12:55,745] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,746] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,747] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,748] - [observetest:156] INFO - ##########key:-observe40################
[2014-08-21 05:12:55,749] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,749] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,750] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,752] - [observetest:156] INFO - ##########key:-observe41################
[2014-08-21 05:12:55,752] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,753] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,754] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,755] - [observetest:156] INFO - ##########key:-observe42################
[2014-08-21 05:12:55,756] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,757] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,757] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,758] - [observetest:156] INFO - ##########key:-observe43################
[2014-08-21 05:12:55,759] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,760] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,760] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,762] - [observetest:156] INFO - ##########key:-observe44################
[2014-08-21 05:12:55,762] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,763] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,763] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,765] - [observetest:156] INFO - ##########key:-observe45################
[2014-08-21 05:12:55,766] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,766] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,767] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,771] - [observetest:156] INFO - ##########key:-observe46################
[2014-08-21 05:12:55,772] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,773] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,774] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,775] - [observetest:156] INFO - ##########key:-observe47################
[2014-08-21 05:12:55,776] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,777] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,777] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,780] - [observetest:156] INFO - ##########key:-observe48################
[2014-08-21 05:12:55,780] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,781] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,782] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,784] - [observetest:156] INFO - ##########key:-observe49################
[2014-08-21 05:12:55,785] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,785] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,786] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,788] - [observetest:156] INFO - ##########key:-observe50################
[2014-08-21 05:12:55,789] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,789] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,790] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,792] - [observetest:156] INFO - ##########key:-observe51################
[2014-08-21 05:12:55,792] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,793] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,794] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,796] - [observetest:156] INFO - ##########key:-observe52################
[2014-08-21 05:12:55,796] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,797] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,798] - [observetest:159] INFO - Time2:- 0:00:00.000017
[2014-08-21 05:12:55,809] - [observetest:156] INFO - ##########key:-observe53################
[2014-08-21 05:12:55,810] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,810] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,811] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,812] - [observetest:156] INFO - ##########key:-observe54################
[2014-08-21 05:12:55,813] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,813] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,814] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,825] - [observetest:156] INFO - ##########key:-observe55################
[2014-08-21 05:12:55,825] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,826] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,826] - [observetest:159] INFO - Time2:- 0:00:00.000023
[2014-08-21 05:12:55,828] - [observetest:156] INFO - ##########key:-observe56################
[2014-08-21 05:12:55,828] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,829] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,829] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,831] - [observetest:156] INFO - ##########key:-observe57################
[2014-08-21 05:12:55,832] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,832] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,833] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,834] - [observetest:156] INFO - ##########key:-observe58################
[2014-08-21 05:12:55,835] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,835] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,836] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,840] - [observetest:156] INFO - ##########key:-observe59################
[2014-08-21 05:12:55,840] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,841] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,842] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,843] - [observetest:156] INFO - ##########key:-observe60################
[2014-08-21 05:12:55,844] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,844] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,845] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,847] - [observetest:156] INFO - ##########key:-observe61################
[2014-08-21 05:12:55,848] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,848] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,849] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,852] - [observetest:156] INFO - ##########key:-observe62################
[2014-08-21 05:12:55,852] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,853] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,854] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,855] - [observetest:156] INFO - ##########key:-observe63################
[2014-08-21 05:12:55,856] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,856] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,857] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:55,858] - [observetest:156] INFO - ##########key:-observe64################
[2014-08-21 05:12:55,859] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,859] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,860] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,862] - [observetest:156] INFO - ##########key:-observe65################
[2014-08-21 05:12:55,862] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,863] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,864] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,874] - [observetest:156] INFO - ##########key:-observe66################
[2014-08-21 05:12:55,874] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,874] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,875] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,876] - [observetest:156] INFO - ##########key:-observe67################
[2014-08-21 05:12:55,877] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,877] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,878] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,880] - [observetest:156] INFO - ##########key:-observe68################
[2014-08-21 05:12:55,881] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,881] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,882] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,884] - [observetest:156] INFO - ##########key:-observe69################
[2014-08-21 05:12:55,885] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,886] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,886] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,890] - [observetest:156] INFO - ##########key:-observe70################
[2014-08-21 05:12:55,890] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,891] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,892] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,895] - [observetest:156] INFO - ##########key:-observe71################
[2014-08-21 05:12:55,895] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,896] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,896] - [observetest:159] INFO - Time2:- 0:00:00.000022
[2014-08-21 05:12:55,897] - [observetest:156] INFO - ##########key:-observe72################
[2014-08-21 05:12:55,898] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,899] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,899] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,901] - [observetest:156] INFO - ##########key:-observe73################
[2014-08-21 05:12:55,902] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,903] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,903] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,905] - [observetest:156] INFO - ##########key:-observe74################
[2014-08-21 05:12:55,905] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,906] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,906] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,908] - [observetest:156] INFO - ##########key:-observe75################
[2014-08-21 05:12:55,909] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,909] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,910] - [observetest:159] INFO - Time2:- 0:00:00.000020
[2014-08-21 05:12:55,911] - [observetest:156] INFO - ##########key:-observe76################
[2014-08-21 05:12:55,912] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,913] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,913] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,916] - [observetest:156] INFO - ##########key:-observe77################
[2014-08-21 05:12:55,916] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,917] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,918] - [observetest:159] INFO - Time2:- 0:00:00.000352
[2014-08-21 05:12:55,919] - [observetest:156] INFO - ##########key:-observe78################
[2014-08-21 05:12:55,919] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,920] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,921] - [observetest:159] INFO - Time2:- 0:00:00.000018
[2014-08-21 05:12:55,922] - [observetest:156] INFO - ##########key:-observe79################
[2014-08-21 05:12:55,923] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,923] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,924] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,926] - [observetest:156] INFO - ##########key:-observe80################
[2014-08-21 05:12:55,927] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,927] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,928] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,930] - [observetest:156] INFO - ##########key:-observe81################
[2014-08-21 05:12:55,931] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,931] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,932] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,934] - [observetest:156] INFO - ##########key:-observe82################
[2014-08-21 05:12:55,935] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,936] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,936] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,938] - [observetest:156] INFO - ##########key:-observe83################
[2014-08-21 05:12:55,939] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,939] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,940] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,942] - [observetest:156] INFO - ##########key:-observe84################
[2014-08-21 05:12:55,942] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,943] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,944] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,945] - [observetest:156] INFO - ##########key:-observe85################
[2014-08-21 05:12:55,946] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,946] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,947] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,948] - [observetest:156] INFO - ##########key:-observe86################
[2014-08-21 05:12:55,949] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,949] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,950] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,951] - [observetest:156] INFO - ##########key:-observe87################
[2014-08-21 05:12:55,952] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,953] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,953] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,955] - [observetest:156] INFO - ##########key:-observe88################
[2014-08-21 05:12:55,955] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,956] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,956] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,958] - [observetest:156] INFO - ##########key:-observe89################
[2014-08-21 05:12:55,958] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,959] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,960] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,961] - [observetest:156] INFO - ##########key:-observe90################
[2014-08-21 05:12:55,961] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,962] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,962] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,964] - [observetest:156] INFO - ##########key:-observe91################
[2014-08-21 05:12:55,964] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,965] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,965] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,967] - [observetest:156] INFO - ##########key:-observe92################
[2014-08-21 05:12:55,967] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,968] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,969] - [observetest:159] INFO - Time2:- 0:00:00.000339
[2014-08-21 05:12:55,970] - [observetest:156] INFO - ##########key:-observe93################
[2014-08-21 05:12:55,971] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,971] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,972] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,973] - [observetest:156] INFO - ##########key:-observe94################
[2014-08-21 05:12:55,974] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,974] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,975] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,977] - [observetest:156] INFO - ##########key:-observe95################
[2014-08-21 05:12:55,978] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,978] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,979] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,984] - [observetest:156] INFO - ##########key:-observe96################
[2014-08-21 05:12:55,984] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,985] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,986] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,989] - [observetest:156] INFO - ##########key:-observe97################
[2014-08-21 05:12:55,990] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,990] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,991] - [observetest:159] INFO - Time2:- 0:00:00.000016
[2014-08-21 05:12:55,992] - [observetest:156] INFO - ##########key:-observe98################
[2014-08-21 05:12:55,992] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,993] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,994] - [observetest:159] INFO - Time2:- 0:00:00.000015
[2014-08-21 05:12:55,995] - [observetest:156] INFO - ##########key:-observe99################
[2014-08-21 05:12:55,995] - [observetest:157] INFO - Persisted:- 1
[2014-08-21 05:12:55,996] - [observetest:158] INFO - Persist_Time:- 0
[2014-08-21 05:12:55,996] - [observetest:159] INFO - Time2:- 0:00:00.000021
[2014-08-21 05:12:56,054] - [rest_client:484] INFO - index query url: http://10.3.5.154:8092/standard_bucket0/_design/dev_Doc1/_view/default_view?stale=false&connection_timeout=60000&full_set=true
ERROR
[('/usr/local/lib/python2.7/threading.py', 783, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/local/lib/python2.7/threading.py', 810, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/local/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/local/lib/python2.7/unittest/case.py', 395, '__call__', 'return self.run(*args, **kwds)'), ('/usr/local/lib/python2.7/unittest/case.py', 331, 'run', 'testMethod()'), ('pytests/observe/observetest.py', 208, 'test_observe_basic_data_load_delete', 'self._run_observe(self)'), ('pytests/observe/observetest.py', 165, '_run_observe', 'self.cluster.query_view(self.master, "dev_Doc1", self.default_view.name, query, self.num_items, bucket, timeout=self.wait_timeout)'), ('lib/couchbase/cluster.py', 464, 'query_view', 'return _task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Thu Aug 21 05:13:55 2014
[2014-08-21 05:13:55,999] - [basetestcase:200] INFO - ==============  basetestcase cleanup was started for test #30 test_observe_basic_data_load_delete ==============
[2014-08-21 05:13:56,030] - [remote_util:142] INFO - connecting to 10.3.5.154 with username : root password : couchbase ssh_key: 
[2014-08-21 05:13:56,081] - [rest_client:1217] INFO - rebalance percentage : 16.80 %
[2014-08-21 05:13:56,200] - [remote_util:174] INFO - Connected to 10.3.5.154
[2014-08-21 05:13:56,345] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: sudo cat /proc/cpuinfo
[2014-08-21 05:13:56,418] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,419] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: df -Th
[2014-08-21 05:13:56,493] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,494] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: sudo cat /proc/meminfo
[2014-08-21 05:13:56,566] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,568] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: hostname
[2014-08-21 05:13:56,640] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,641] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: hostname -d
[2014-08-21 05:13:56,753] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,755] - [remote_util:1531] INFO - running command.raw on 10.3.5.154: killall -9 memcached
[2014-08-21 05:13:56,833] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:56,836] - [remote_util:142] INFO - connecting to 10.3.5.155 with username : root password : couchbase ssh_key: 
[2014-08-21 05:13:57,084] - [remote_util:174] INFO - Connected to 10.3.5.155
[2014-08-21 05:13:57,253] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: sudo cat /proc/cpuinfo
[2014-08-21 05:13:57,303] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,303] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: df -Th
[2014-08-21 05:13:57,379] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,380] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: sudo cat /proc/meminfo
[2014-08-21 05:13:57,452] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,453] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: hostname
[2014-08-21 05:13:57,526] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,527] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: hostname -d
[2014-08-21 05:13:57,578] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,580] - [remote_util:1531] INFO - running command.raw on 10.3.5.155: killall -9 memcached
[2014-08-21 05:13:57,658] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,661] - [remote_util:142] INFO - connecting to 10.3.2.146 with username : root password : couchbase ssh_key: 
[2014-08-21 05:13:57,818] - [remote_util:174] INFO - Connected to 10.3.2.146
[2014-08-21 05:13:57,945] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: sudo cat /proc/cpuinfo
[2014-08-21 05:13:57,995] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:57,996] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: df -Th
[2014-08-21 05:13:58,069] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:58,069] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: sudo cat /proc/meminfo
[2014-08-21 05:13:58,141] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:58,141] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: hostname
[2014-08-21 05:13:58,188] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:58,189] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: hostname -d
[2014-08-21 05:13:58,263] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:58,265] - [remote_util:1531] INFO - running command.raw on 10.3.2.146: killall -9 memcached
[2014-08-21 05:13:58,340] - [remote_util:1560] INFO - command executed successfully
[2014-08-21 05:13:58,343] - [basetestcase:211] WARNING - rebalancing is still running, test should be verified
[2014-08-21 05:13:58,358] - [rest_client:1646] INFO - rebalance was stopped
[2014-08-21 05:13:58,476] - [bucket_helper:137] INFO - deleting existing buckets [u'bucket0', u'default', u'standard_bucket0'] on 10.3.5.154
[2014-08-21 05:13:58,477] - [bucket_helper:139] INFO - remove bucket bucket0 ...
[2014-08-21 05:14:08,100] - [rest_client:1200] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2014-08-21 05:14:08,138] - [bucket_helper:153] INFO - deleted bucket : bucket0 from 10.3.5.154
[2014-08-21 05:14:08,144] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
[2014-08-21 05:14:08,145] - [rest_client:2010] INFO - Latest logs from UI on 10.3.5.154:
[2014-08-21 05:14:08,146] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 0, u'text': u'Bucket "standard_bucket0" loaded on node \'ns_1@10.3.5.154\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-08-21T05:18:04.387Z', u'module': u'ns_memcached', u'tstamp': 1408623484387, u'type': u'info'}
[2014-08-21 05:14:08,147] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.155', u'code': 0, u'text': u'Bucket "standard_bucket0" loaded on node \'ns_1@10.3.5.155\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-08-21T05:18:02.170Z', u'module': u'ns_memcached', u'tstamp': 1408623482170, u'type': u'info'}
[2014-08-21 05:14:08,147] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.154\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-08-21T05:18:02.007Z', u'module': u'ns_memcached', u'tstamp': 1408623482007, u'type': u'info'}
[2014-08-21 05:14:08,148] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 0, u'text': u'Shutting down bucket "bucket0" on \'ns_1@10.3.5.154\' for deletion', u'shortText': u'message', u'serverTime': u'2014-08-21T05:18:02.006Z', u'module': u'ns_memcached', u'tstamp': 1408623482006, u'type': u'info'}
[2014-08-21 05:14:08,148] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.155', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.155\' in 1 seconds.', u'shortText': u'message', u'serverTime': u'2014-08-21T05:18:01.665Z', u'module': u'ns_memcached', u'tstamp': 1408623481665, u'type': u'info'}
[2014-08-21 05:14:08,149] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.155', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@10.3.5.155' disconnected: {badmatch,\n                                                                    {error,\n                                                                     closed}}", u'shortText': u'message', u'serverTime': u'2014-08-21T05:17:57.672Z', u'module': u'ns_memcached', u'tstamp': 1408623477672, u'type': u'info'}
[2014-08-21 05:14:08,150] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.155', u'code': 0, u'text': u"Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 137. Restarting. Messages: Thu Aug 21 05:17:56.807692 PDT 3: (standard_bucket0) DCP (Producer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 630) Stream closing, 0 items sent from disk, 0 items sent from memory, 0 was last seqno sent\nThu Aug 21 05:17:56.807706 PDT 3: (standard_bucket0) DCP (Producer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 631) Stream closing, 0 items sent from disk, 0 items sent from memory, 0 was last seqno sent\nThu Aug 21 05:17:56.807721 PDT 3: (standard_bucket0) DCP (Producer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 632) Stream closing, 0 items sent from disk, 0 items sent from memory, 0 was last seqno sent\nThu Aug 21 05:17:56.807735 PDT 3: (standard_bucket0) DCP (Producer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 633) Stream closing, 0 items sent from disk, 0 items sent from memory, 0 was last seqno sent\nThu Aug 21 05:17:56.807749 PDT 3: (standard_bucket0) DCP (Producer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 634) Stream closing, 0 items sent from disk, 0 items sent from memory, 0 was last seqno sent", u'shortText': u'message', u'serverTime': u'2014-08-21T05:17:57.492Z', u'module': u'ns_log', u'tstamp': 1408623477492, u'type': u'info'}
[2014-08-21 05:14:08,150] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n                              {'EXIT',<0.32255.122>,\n                               {{{badmatch,{error,closed}},\n                                 {gen_server,call,\n                                  ['ns_memcached-standard_bucket0',\n                                   {set_vbucket,610,replica},\n                                   180000]}},\n                                {gen_server,call,\n                                 [{'janitor_agent-standard_bucket0',\n                                   'ns_1@10.3.5.154'},\n                                  {if_rebalance,<0.14541.122>,\n                                   {dcp_takeover,'ns_1@10.3.5.155',634}},\n                                  infinity]}}}}\n", u'shortText': u'message', u'serverTime': u'2014-08-21T05:17:57.340Z', u'module': u'ns_orchestrator', u'tstamp': 1408623477340, u'type': u'info'}
[2014-08-21 05:14:08,151] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 0, u'text': u"<0.31685.122> exited with {unexpected_exit,\n                           {'EXIT',<0.32255.122>,\n                            {{{badmatch,{error,closed}},\n                              {gen_server,call,\n                               ['ns_memcached-standard_bucket0',\n                                {set_vbucket,610,replica},\n                                180000]}},\n                             {gen_server,call,\n                              [{'janitor_agent-standard_bucket0',\n                                'ns_1@10.3.5.154'},\n                               {if_rebalance,<0.14541.122>,\n                                {dcp_takeover,'ns_1@10.3.5.155',634}},\n                               infinity]}}}}", u'shortText': u'message', u'serverTime': u'2014-08-21T05:17:57.317Z', u'module': u'ns_vbucket_mover', u'tstamp': 1408623477317, u'type': u'critical'}
[2014-08-21 05:14:08,151] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.154', u'code': 0, u'text': u"Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 137. Restarting. Messages: Thu Aug 21 05:17:55.949046 PDT 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 638) Attempting to add takeover stream with start seqno 0, end seqno 18446744073709551615, vbucket uuid 19403751309640, snap start seqno 0, and snap end seqno 0\nThu Aug 21 05:17:56.087632 PDT 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 637) Attempting to add takeover stream with start seqno 2, end seqno 18446744073709551615, vbucket uuid 147423988606456, snap start seqno 2, and snap end seqno 2\nThu Aug 21 05:17:56.320087 PDT 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 636) Attempting to add takeover stream with start seqno 0, end seqno 18446744073709551615, vbucket uuid 230354286269916, snap start seqno 0, and snap end seqno 0\nThu Aug 21 05:17:56.584387 PDT 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 635) Attempting to add takeover stream with start seqno 2, end seqno 18446744073709551615, vbucket uuid 201592126763244, snap start seqno 2, and snap end seqno 2\nThu Aug 21 05:17:57.252362 PDT 3: (standard_bucket0) DCP (Consumer) eq_dcpq:replication:ns_1@10.3.5.155->ns_1@10.3.5.154:standard_bucket0 - (vb 634) Attempting to add takeover stream with start seqno 0, end seqno 18446744073709551615, vbucket uuid 273841142579945, snap start seqno 0, and snap end seqno 0", u'shortText': u'message', u'serverTime': u'2014-08-21T05:17:57.313Z', u'module': u'ns_log', u'tstamp': 1408623477313, u'type': u'info'}
[('/usr/local/lib/python2.7/threading.py', 783, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/local/lib/python2.7/threading.py', 810, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 58, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 370, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Thu Aug 21 05:14:08 2014
[2014-08-21 05:14:08,230] - [rest_client:122] INFO - existing buckets : [u'default', u'standard_bucket0']
[2014-08-21 05:14:08,231] - [bucket_helper:139] INFO - remove bucket default ...
[2014-08-21 05:14:09,581] - [bucket_helper:153] INFO - deleted bucket : default from 10.3.5.154
[2014-08-21 05:14:09,582] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
[2014-08-21 05:14:09,621] - [rest_client:122] INFO - existing buckets : [u'standard_bucket0']
[2014-08-21 05:14:09,622] - [bucket_helper:139] INFO - remove bucket standard_bucket0 ...
[2014-08-21 05:14:10,820] - [bucket_helper:153] INFO - deleted bucket : standard_bucket0 from 10.3.5.154
[2014-08-21 05:14:10,821] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
[2014-08-21 05:14:10,832] - [rest_client:122] INFO - existing buckets : []
[2014-08-21 05:14:10,850] - [bucket_helper:137] INFO - deleting existing buckets [] on 10.3.5.155
[2014-08-21 05:14:10,861] - [bucket_helper:137] INFO - deleting existing buckets [] on 10.3.2.146
[2014-08-21 05:14:10,894] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
[2014-08-21 05:14:10,900] - [rest_client:1095] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.5.155&user=Administrator&knownNodes=ns_1%4010.3.5.154%2Cns_1%4010.3.5.155
[2014-08-21 05:14:10,907] - [rest_client:1099] INFO - rebalance operation started
[2014-08-21 05:14:10,915] - [rest_client:1217] INFO - rebalance percentage : 0.00 %
[2014-08-21 05:14:20,936] - [rest_client:1172] INFO - rebalance progress took 10.028897047 seconds 
[2014-08-21 05:14:20,937] - [rest_client:1173] INFO - sleep for 10 seconds after rebalance...
[2014-08-21 05:14:30,963] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:10.3.5.154 port:8091 ssh_username:root ? [(u'ns_1@10.3.5.155', 8091)]
[2014-08-21 05:14:30,963] - [basetestcase:264] INFO - sleep for 10 secs.  ...
[2014-08-21 05:14:40,982] - [cluster_helper:78] INFO - waiting for ns_server @ 10.3.5.154:8091
[2014-08-21 05:14:40,993] - [cluster_helper:80] INFO - ns_server @ 10.3.5.154:8091 is running
[2014-08-21 05:14:41,000] - [cluster_helper:78] INFO - waiting for ns_server @ 10.3.5.155:8091
[2014-08-21 05:14:41,007] - [cluster_helper:80] INFO - ns_server @ 10.3.5.155:8091 is running
[2014-08-21 05:14:41,014] - [cluster_helper:78] INFO - waiting for ns_server @ 10.3.2.146:8091
[2014-08-21 05:14:41,021] - [cluster_helper:80] INFO - ns_server @ 10.3.2.146:8091 is running
[2014-08-21 05:14:41,022] - [basetestcase:219] INFO - ==============  basetestcase cleanup was finished for test #30 test_observe_basic_data_load_delete ==============

======================================================================
ERROR: test_observe_basic_data_load_delete (observe.observetest.ObserveTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/observe/observetest.py", line 208, in test_observe_basic_data_load_delete
    self._run_observe(self)
  File "pytests/observe/observetest.py", line 165, in _run_observe
    self.cluster.query_view(self.master, "dev_Doc1", self.default_view.name, query, self.num_items, bucket, timeout=self.wait_timeout)
  File "lib/couchbase/cluster.py", line 464, in query_view
    return _task.result(timeout)
  File "lib/tasks/future.py", line 163, in result
    raise TimeoutError()
TimeoutError

----------------------------------------------------------------------
Ran 1 test in 297.353s