Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: backlog
Affects Version/s: 6.5.0
Component/s: ns_server
Labels:

Triage:
Triaged
Is this a Regression?:
No
Sprint:
KV Sprint 2019-12

Description

Build: 6.5.0-4908
Test: -test tests/2i/test_idx_rebalance_replica_mad_hatter.yml -scope tests/2i/scope_idx_rebalance_replica_12nodes.yml
Scale: 2

172.23.96.18

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.456-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 1 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.456-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:08.463-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 1 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:09.096-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 2 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:09.759-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 3 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:10.451-08:00 [Info] StorageMgr::handleRollback Rollback Index: 10744950810890384087 PartitionId: 4 SliceId: 0 To Zero

172.23.96.210

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.322-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8063402928482161013 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.322-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:29.334-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8063402928482161013 PartitionId: 0 SliceId: 0 To Zero

172.23.96.212

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.111-08:00 [Info] StorageMgr::handleRollback Rollback Index: 7738569596459039496 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.111-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM other-2

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:33.129-08:00 [Info] StorageMgr::handleRollback Rollback Index: 7738569596459039496 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.470-08:00 [Info] StorageMgr::handleRollback Rollback Index: 15572186898635795728 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.470-08:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM default

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:50:48.479-08:00 [Info] StorageMgr::handleRollback Rollback Index: 15572186898635795728 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:01.053-08:00 [Info] StorageMgr::handleRollback Rollback Index: 13528572186754933381 PartitionId: 0 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:05.824-08:00 [Info] StorageMgr::handleRollback Rollback Index: 3100293769907018678 PartitionId: 4 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:08.724-08:00 [Info] StorageMgr::handleRollback Rollback Index: 3666011380695847645 PartitionId: 2 SliceId: 0 To Zero

        /opt/couchbase/var/lib/couchbase/logs/indexer.log:2019-12-05T07:51:28.077-08:00 [Info] StorageMgr::handleRollback Rollback Index: 8281259647172631345 PartitionId: 0 SliceId: 0 To Zero

Memcached is killed in 172.23.96.48

        /opt/couchbase/var/lib/couchbase/logs/info.log.1.gz:[user:info,2019-12-05T07:47:45.665-08:00,ns_1@172.23.96.48:<0.12168.0>:ns_log:crash_consumption_loop:69]Service 'memcached' exited with status 137. Restarting. Messages:

Test step:

*[2019-12-05T07:47:43-08:00, vijayviji/sshpass:b8f3af] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.96.48 kill -SIGKILL $(pgrep memcached)*

[2019-12-05T07:47:48-08:00, sequoiatools/cmd:744a58] 600

[2019-12-05T07:58:03-08:00, sequoiatools/couchbase-cli:6.5:c96cc6] server-add -c 172.23.97.74:8091 --server-add https://172.23.96.48 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.183.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.191.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.96.212.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1575561371/collectinfo-2019-12-05T155613-ns_1%40172.23.97.74.zip

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[System test] rollbackAllToZero when memcached is killed

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty