Loading...

Details

Type: Bug
Resolution: Not a Bug
Priority: Critical
Fix Version/s: 7.0.2
Affects Version/s: Cheshire-Cat
Component/s: secondary-index, storage-engine
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.15.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.165.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.214.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.244.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.102.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.118.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.164.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.183.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.206.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.29.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.62.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.90.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.93.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.191.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.207.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.209.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.225.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.70.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.15.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.165.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.214.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.232.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.244.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.104.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.102.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.109.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.118.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.164.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.183.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.206.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.210.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.29.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.61.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.62.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.90.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.105.93.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.117.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.191.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.207.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.209.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.225.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.232.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.54.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614813105/collectinfo-2021-03-03T231147-ns_1%40172.23.106.70.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-4587
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 2
Iteration : 1st

The test started rebalancing out an analytics node 172.23.105.164 at 2021-03-03T13:23:39. On 13:52:01, indexer on 172.23.106.54 crashed with status 133. The following could be found in the indexer logs :

2021-03-03T13:52:01.278-08:00 [Info] Indexer::handleDropIndex - IndexInstId 9713538497774980479

2021-03-03T13:52:01.279-08:00 [Info] MutationMgr::handleUpdateIndexInstMap, updated instances:

        InstanceId: 9713538497774980479 Name: idx3_RdrXAm Keyspace: bucket4:scope_0:coll_0 State: INDEX_STATE_DELETED Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 2

2021-03-03T13:52:01.279-08:00 [Info] ScanCoordinator::initialize rollback times on new index inst map: map[bucket4:1614792564360261132 bucket5:1614792909036460638 bucket6:1614793262197204729 bucket7:1614793665469679930 default:1614789429479736463]

2021-03-03T13:52:01.279-08:00 [Info] ClustMgr::handleIndexMap, updated instances:

        InstanceId: 9713538497774980479 Name: idx3_RdrXAm Keyspace: bucket4:scope_0:coll_0 State: INDEX_STATE_DELETED Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 2

2021-03-03T13:52:01.280-08:00 [Info] MutationMgr::handleUpdateIndexInstMap, deleted instance id's: [9713538497774980479]

2021-03-03T13:52:01.280-08:00 [Info] MutationMgr::handleUpdateIndexPartnMap, deleted instance id: 9713538497774980479

2021-03-03T13:52:01.280-08:00 [Info] ScanCoordinator::initialize rollback times on new index inst map: map[bucket4:1614792564360261132 bucket5:1614792909036460638 bucket6:1614793262197204729 bucket7:1614793665469679930 default:1614789429479736463]

2021-03-03T13:52:01.280-08:00 [Info] ClustMgr::handleIndexMap, deleted instance id's: [9713538497774980479]

2021-03-03T13:52:01.280-08:00 [Info] plasmaSlice::Close Closing Slice Id 0, IndexInstId 9713538497774980479, PartitionId 2, IndexDefnId 4589627384111031498

2021-03-03T13:52:01.280-08:00 [Info] Shard /data/couchbase/@2i/shards/shard4(4) : Instance removed from swapper : /data/couchbase/@2i/shards/shard4

2021-03-03T13:52:01.280-08:00 [Info] Shard /data/couchbase/@2i/shards/shard4(4) : instance /data/couchbase/@2i/bucket4_idx3_RdrXAm_9713538497774980479_2.index/mainIndex stopped

fatal: morestack on gsignal

Initializing write barrier = 8000

2021-03-03T13:52:02.061-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/couchbase/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=c2462c8c42c1ef32a0a4913a4c01c6f4 -ipv6=false -isEnterprise=true]

2021-03-03T13:52:02.071-08:00 [Info] Indexer::NewIndexer Status Warmup

Disable Breakpad integration in ForestDB (delete)

Details

Description

Attachments

Activity

People

Dates

PagerDuty