Major Description
This is seen while running a swap/rebalance test but actually occurs before any swap/rebalance happens, there is memcache crash while creating the bucket.
The test is:
python testrunner.py -i ~/servers/tunable-vms.ini max-dupe-result-count=10,get-cbcollect-info=False,num-tries=60,attempt-num=60,dgm_run=true-t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=1,num-buckets=1,num-swap=3,percentage_progress=40
|
And the test runner output at the time of the issue is:
2016-08-30 12:25:05 | INFO | MainProcess | test_thread | [swaprebalance._common_test_body_failed_swap_rebalance] CREATE BUCKET PHASE
|
2016-08-30 12:25:05 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.24 existing buckets : []
|
2016-08-30 12:25:05 | INFO | MainProcess | test_thread | [rest_client.create_bucket] http://172.23.106.24:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1342&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=
|
2016-08-30 12:25:05 | INFO | MainProcess | test_thread | [rest_client.create_bucket] 0.06 seconds to create bucket default
|
2016-08-30 12:25:05 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.106.24 to accept set ops
|
2016-08-30 12:26:05 | INFO | MainProcess | test_thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
|
|
2016-08-30 12:27:06 | INFO | MainProcess | test_thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
The logs show:
Service 'memcached' exited with status 134. Restarting. Messages: 2016-08-30T13:22:33.459475-07:00 WARNING /opt/couchbase/bin/memcached() [0x413b69]
|
2016-08-30T13:22:33.459506-07:00 WARNING /opt/couchbase/bin/../lib/libplatform.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0x1b) [0x7fba4c0aea6b]
|
2016-08-30T13:22:33.459531-07:00 WARNING /opt/couchbase/bin/../lib/libplatform.so.0.1.0() [0x7fba4c0aa75a]
|
2016-08-30T13:22:33.459589-07:00 WARNING /lib64/libpthread.so.0() [0x7fba4bc8cdc5]
|
2016-08-30T13:22:33.459682-07:00 WARNING /lib64/libc.so.6(clone+0x6d) [0x7fba49c9d21d]
|
This does not happen for the usual Spock build
The logs are attached as is the .ini file used.
Please let me know if there more information I can provide.