Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
6.5.0
-
Triaged
-
No
-
KV Sprint 2019-12
Description
Build: 6.5.0-4908
Test: -test tests/2i/test_idx_rebalance_replica_mad_hatter.yml -scope tests/2i/scope_idx_rebalance_replica_12nodes.yml
Scale: 2
172.23.96.18
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.456-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 1 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.456-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.463-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 1 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:09.096-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 2 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:09.759-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 3 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:10.451-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 4 SliceId: 0 To Zero
|
172.23.96.210
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.322-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8063402928482161013 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.322-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.334-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8063402928482161013 PartitionId: 0 SliceId: 0 To Zero
|
172.23.96.212
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.111-08:00 [Info] StorageMgr::handleRollback Rollback Index: 7738569596459039496 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.111-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.129-08:00 [Info] StorageMgr::handleRollback Rollback Index: 7738569596459039496 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.470-08:00 [Info] StorageMgr::handleRollback Rollback Index: 15572186898635795728 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.470-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM default
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.479-08:00 [Info] StorageMgr::handleRollback Rollback Index: 15572186898635795728 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:01.053-08:00 [Info] StorageMgr::handleRollback Rollback Index: 13528572186754933381 PartitionId: 0 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:05.824-08:00 [Info] StorageMgr::handleRollback Rollback Index: 3100293769907018678 PartitionId: 4 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:08.724-08:00 [Info] StorageMgr::handleRollback Rollback Index: 3666011380695847645 PartitionId: 2 SliceId: 0 To Zero
|
/opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:28.077-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8281259647172631345 PartitionId: 0 SliceId: 0 To Zero
|
Memcached is killed in 172.23.96.48
/opt/couchbase/var/lib/couchbase/logs/info.log.1.gz:[user:info,2019-12-05T07:47:45.665-08:00,ns_1@172.23.96.48:<0.12168.0>:ns_log:crash_consumption_loop:69]Service 'memcached' exited with status 137. Restarting. Messages:
|
Test step:
*[2019-12-05T07:47:43-08:00, vijayviji/sshpass:b8f3af] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.96.48 kill -SIGKILL $(pgrep memcached)*
|
[2019-12-05T07:47:48-08:00, sequoiatools/cmd:744a58] 600
|
[2019-12-05T07:58:03-08:00, sequoiatools/couchbase-cli:6.5:c96cc6] server-add -c 172.23.97.74:8091 --server-add https://172.23.96.48 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.183.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.191.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.212.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.97.74.zip