Details
-
Bug
-
Resolution: User Error
-
Critical
-
Cheshire-Cat
-
7.0.0-5219
-
Untriaged
-
-
1
-
Unknown
Description
Steps:
1. Create a 6 node cluster:
+----------------+-----------+-----------------------+---------------+--------------+
|
| Nodes | Services | Version | CPU | Status |
|
+----------------+-----------+-----------------------+---------------+--------------+
|
| 172.23.106.71 | kv | 7.0.0-5219-enterprise | 1.48036632794 | Cluster node |
|
| 172.23.106.216 | ['kv'] | | | <--- IN --- |
|
| 172.23.106.215 | ['kv'] | | | <--- IN --- |
|
| 172.23.106.206 | ['kv'] | | | <--- IN --- |
|
| 172.23.106.211 | ['n1ql'] | | | <--- IN --- |
|
| 172.23.106.214 | ['index'] | | | <--- IN --- |
|
+----------------+-----------+-----------------------+---------------+--------------+
|
2. Create a default magma bucket. Load some initial data in the bucket.
3. Start a new data loading thread.
4. Start a crash thread that is kill -9 memcached on data nodes. In every kill, test wait for bucket to come online. To check that we run cbstats continuously on the nodes to get the warmup=complete status.
5. While cbstats is running, bucket is stuck in warmup and not able to come out of it. As soon as cbstats stopped bucket status becomes green.
Node 71 cbstats |
2021-05-24 07:14:23,694 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:3893] Exception during cbstat all cmd: Traceback (most recent call last):
|
|
File "/opt/couchbase/lib/python/cbstats", line 1010, in <module>
|
|
main()
|
|
File "/opt/couchbase/lib/python/cbstats", line 1007, in main
|
|
c.execute()
|
|
File "/opt/couchbase/lib/python/clitool.py", line 83, in execute
|
|
f[0](mc, *args[2:], **opts.__dict__)
|
|
File "/opt/couchbase/lib/python/cbstats", line 49, in g
|
|
f(*args, **kwargs)
|
|
File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g
|
|
mc.sasl_auth_plain(username, password)
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain
|
|
return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start
|
|
return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd
|
|
return self._handleSingleResponse(opaque)
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse
|
|
cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse
|
|
cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg()
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg
|
|
data = self._socketRecv(MIN_RECV_PACKET - len(response))
|
|
File "/opt/couchbase/lib/python/mc_bin_client.py", line 245, in _socketRecv
|
|
raise TimeoutError(30)
|
|
mc_bin_client.TimeoutError: Error: Operation timed out (30 seconds)
|
Note: Not seen this issue on couchstore but this issue seems to be occurring on magma very recently. Not quite sure on the regression build. A quick analysis on the issue would be helpful.
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.10991.ini bucket_storage=magma,rerun=false,GROUP=P1;recovery,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,replicas=0,nodes_init=4,dcp_services=n1ql-index,enable_dp=True,collect_pcaps=True,transaction_version=1.1.5,upgrade_version=7.0.0-5219 -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery,num_items=10000000,doc_size=1024,sdk_timeout=60,doc_ops=create:delete,replicas=0,GROUP=P1;recovery'
|
Test Run(test_1): http://qa.sc.couchbase.com/job/test_suite_executor-TAF/116874/console
Attachments
Issue Links
- is duplicated by
-
MB-45330 [Magma] After restarting the couchbase server, observed "ERROR 2019: Create bucket [default] failed - Already exists"
- Closed