Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
4.6.0, 5.0.0
-
Untriaged
-
Yes
Description
4.7.0 Build : 1251(http://server.jenkins.couchbase.com/job/build_sanity_matrix/2175/)
4.6.0 Build : 3405 (http://server.jenkins.couchbase.com/job/build_sanity_matrix/2171/)
Observing that the following tests intermittently in 4.6.0 and 4.7.0 builds
drainratetests.DrainRateTests.test_drain_100k_items
This failure is not seen on all Distros, but is prominently seen on CentOS7 1-node platform, especially for 4.7.0.
Triaged the failure in http://server.jenkins.couchbase.com/job/build_sanity_matrix/2175/DISTRO=centos7,TYPE=1node/consoleFull.
The failure occurs in the test at 2016-10-20 21:50:20. There is a crash reported in the ns_server logs around the same time - 2016-10-20T21:50:17, and subsequently all services restart.
Verified this behaviour in a couple of test runs where the test have failed.
Test console output -
21:50:17 [2016-10-20 21:50:17,928] - [rest_client:1921] INFO - http://172.23.107.190:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=744&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword= |
21:50:17 [2016-10-20 21:50:17,934] - [rest_client:1943] INFO - 0.01 seconds to create bucket default |
21:50:17 [2016-10-20 21:50:17,935] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 172.23.107.190 to accept set ops |
21:50:20 [2016-10-20 21:50:20,072] - [data_helper:295] INFO - creating direct client 172.23.107.190:11210 default |
21:50:20 [2016-10-20 21:50:20,122] - [data_helper:825] WARNING - unable to establish connection to 172.23.107.190. cleanup open connections |
21:50:20 ERROR |
21:50:20 |
21:50:20 ====================================================================== |
21:50:20 ERROR: test_drain_100k_items (drainratetests.DrainRateTests) |
21:50:20 ---------------------------------------------------------------------- |
21:50:20 Traceback (most recent call last): |
21:50:20 File "pytests/drainratetests.py", line 20, in setUp |
21:50:20 self._create_default_bucket() |
21:50:20 File "pytests/drainratetests.py", line 57, in _create_default_bucket |
21:50:20 ready = BucketOperationHelper.wait_for_memcached(master, name) |
21:50:20 File "lib/membase/helper/bucket_helper.py", line 324, in wait_for_memcached |
21:50:20 bucket, timeout_in_seconds, log_msg) |
21:50:20 File "lib/membase/helper/bucket_helper.py", line 266, in wait_for_vbuckets_ready_state |
21:50:20 obj = VBucketAwareMemcached(rest, bucket) |
21:50:20 File "lib/memcached/helper/data_helper.py", line 741, in __init__ |
21:50:20 self.reset(rest) |
21:50:20 File "lib/memcached/helper/data_helper.py", line 746, in reset |
21:50:20 m, v, r = self.request_map(self.rest , self.bucket) |
21:50:20 File "lib/memcached/helper/data_helper.py", line 797, in request_map |
21:50:20 self.add_memcached(vBucket.master, memcacheds, rest, bucket) |
21:50:20 File "lib/memcached/helper/data_helper.py", line 827, in add_memcached |
21:50:20 raise ex |
21:50:20 MemcachedError: Memcached error #32 'Auth error': Auth failure for vbucket :0 to mc 172.23.107.190:11210 |
21:50:20 |
21:50:20 ---------------------------------------------------------------------- |