Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
6.6.3
-
6.6.2-9588 -> 6.6.3-9799
-
Untriaged
-
Centos 64-bit
-
1
-
Yes
Description
Steps to Repro
1. Run the following 6.6.2 longevity test for 4 days.
./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=true -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. At this point we would have 27 node cluster(3 analytics, 3 index, 3 fts, 3 query, 6 index, 9 data).
3. Failover(graceful for kv) 6 nodes (one of each service type), upgrade and do a recovery(delta for kv) and rebalance.
4. Rebalance completes successfully.
However I see lot of messages like below in the logs.
172.23.110.75 : crash
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T02:56:12.577-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T02:57:11.849-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T02:58:11.418-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T02:59:10.670-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:00:09.990-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:01:09.263-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:02:08.933-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:03:08.326-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:04:07.584-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:05:07.044-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:06:06.450-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:07:05.707-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:08:04.966-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:09:04.235-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:10:03.491-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-07-23T03:11:02.844-07:00,ns_1@172.23.110.75:<0.579.0>:ns_log:crash_consumption_loop:69]Service 'cbas' exited with status 1. Restarting. Messages:
|
Note
1. I had one one more upgrade previously from 6.6.2-9588 -> 6.6.3-9796 when this did not happen. So marking it regression.
2. This does not affect our testing in any way.
cbcollect_info attached. Looks similar to MB-45648.