Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0-beta
-
Security Level: Public
-
None
-
0.75
Description
After getting list of processes from node I'm seeing that couch_server is doing sync db deletion notification, which is being delivered to set_views thing, which is doing some call to replica index group, which tries to open db (calls back to couch_server).
Entire thing is stuck during bucket deletion where after shutting down bucket in ep-engine we're deleting vbuckets. In this case we're stuck deleting master vbucket.
Original description follows:
http://qa.hq.northscale.net/job/ubuntu-64-2.0-single-node/119/console
build 2.0.0-1579-rel
tests creates some diff buckets and then tries to delete them
when we try to delete buckets we get responses in logs: 500 reason: unknown
[2012-08-13 10:44:22,948] - [bucket_helper:124] INFO - deleting existing buckets on [ip:10.3.3.20 port:8091 ssh_username:root]
[2012-08-13 10:44:53,646] - [rest_client:561] ERROR - http://10.3.3.20:8091//pools/default/buckets/default error 500 reason: unknown
[2012-08-13 10:44:53,646] - [bucket_helper:136] INFO - deleted bucket : default from 10.3.3.20
[2012-08-13 10:44:53,647] - [bucket_helper:161] INFO - waiting for bucket deletion to complete....
[2012-08-13 10:44:54,051] - [rest_client:96] INFO - existing buckets : [u'set-get-bucket-replica-1', u'set-get-bucket-replica-2', u'set-get-bucket-replica-3']
[2012-08-13 10:45:24,083] - [rest_client:561] ERROR - http://10.3.3.20:8091//pools/default/buckets/set-get-bucket-replica-1 error 500 reason: unknown
[2012-08-13 10:45:24,084] - [bucket_helper:136] INFO - deleted bucket : set-get-bucket-replica-1 from 10.3.3.20
[2012-08-13 10:45:24,084] - [bucket_helper:161] INFO - waiting for bucket deletion to complete....
[2012-08-13 10:45:24,396] - [rest_client:96] INFO - existing buckets : [u'set-get-bucket-replica-2', u'set-get-bucket-replica-3']
[2012-08-13 10:45:54,416] - [rest_client:561] ERROR - http://10.3.3.20:8091//pools/default/buckets/set-get-bucket-replica-2 error 500 reason: unknown
[2012-08-13 10:45:54,417] - [bucket_helper:136] INFO - deleted bucket : set-get-bucket-replica-2 from 10.3.3.20
[2012-08-13 10:45:54,417] - [bucket_helper:161] INFO - waiting for bucket deletion to complete....
[2012-08-13 10:45:54,595] - [rest_client:96] INFO - existing buckets : [u'set-get-bucket-replica-3']
[2012-08-13 10:46:24,619] - [rest_client:561] ERROR - http://10.3.3.20:8091//pools/default/buckets/set-get-bucket-replica-3 error 500 reason: unknown
[2012-08-13 10:46:24,619] - [bucket_helper:136] INFO - deleted bucket : set-get-bucket-replica-3 from 10.3.3.20
[2012-08-13 10:46:24,620] - [bucket_helper:161] INFO - waiting for bucket deletion to complete....
[2012-08-13 10:46:24,627] - [rest_client:96] INFO - existing buckets : []
ok
----------------------------------------------------------------------
Ran 1 test in 179.644s
OK
test_expired_keys (expirytests.ExpiryTests) ... summary so far suite recreatebuckettests.RecreateMembaseBuckets , pass 3 , fail 0
summary so far suite deletebuckettests.DeleteMembaseBuckets , pass 2 , fail 0
summary so far suite createbuckettests.CreateMembaseBucketsTests , pass 16 , fail 0
summary so far suite setgettests.MembaseBucket , pass 1 , fail 0
testrunner logs, diags and results are available under logs/testrunner-12-Aug-13_10-35-44
./testrunner -i onenode.ini -t expirytests.ExpiryTests.test_expired_keys
[Test Input params] :
[2012-08-13 10:46:24,708] - [bucket_helper:124] INFO - deleting existing buckets on [ip:10.3.3.20 port:8091 ssh_username:root]
[2012-08-13 10:46:24,760] - [rest_client:580] INFO - settings/web params : username=Administrator&password=password&port=8091
[2012-08-13 10:46:24,768] - [rest_client:606] INFO - pools/default params : memoryQuota=1607&username=Administrator&password=password
[2012-08-13 10:46:24,774] - [rest_client:1222] INFO - http://10.3.3.20:8091//pools/default/buckets with param: proxyPort=11210&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=&replicaNumber=1&ramQuotaMB=1071
[2012-08-13 10:46:24,789] - [rest_client:561] ERROR - http://10.3.3.20:8091//pools/default/buckets error 503 reason: unknown
[2012-08-13 10:46:24,789] - [rest_client:1227] INFO - 0.0152242183685 seconds to create bucket default
ERROR
======================================================================
ERROR: test_expired_keys (expirytests.ExpiryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "pytests/expirytests.py", line 42, in setUp
proxyPort=info.memcached)
File "lib/membase/api/rest_client.py", line 1230, in create_bucket
raise BucketCreationException(ip=self.ip, bucket_name=bucket)
BucketCreationException: unable to create bucket default on the host @ 10.3.3.20