Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
400-3502
▪ Machine configuration:
⁃ SSDs
⁃ 24GB Ram
⁃ 250GB /data and /index disks
⁃ 8 Core CPU
⁃ Individual VMs ( non-shared)
▪ Indexer Settings : ( see attached settings.json)
⁃ Index Ram Quota on Index Nodes : 22 GB,
⁃ 8 Indexer Threads
⁃ indexer.settings.scan_timeout:1400000 No index settings changed at any point on the test.
▪ 8 Node cluster
⁃ 2 Index nodes,
⁃ 2 Query nodes,
⁃ 4 KV nodes
▪ 10 Buckets
▪ 50 warehouses for tppc(9) buckets and 5M items for default.
▪ 18 GSI indexes across 10 buckets
400-3502 ▪ Machine configuration: ⁃ SSDs ⁃ 24GB Ram ⁃ 250GB /data and /index disks ⁃ 8 Core CPU ⁃ Individual VMs ( non-shared) ▪ Indexer Settings : ( see attached settings.json) ⁃ Index Ram Quota on Index Nodes : 22 GB, ⁃ 8 Indexer Threads ⁃ indexer.settings.scan_timeout:1400000 No index settings changed at any point on the test. ▪ 8 Node cluster ⁃ 2 Index nodes, ⁃ 2 Query nodes, ⁃ 4 KV nodes ▪ 10 Buckets ▪ 50 warehouses for tppc(9) buckets and 5M items for default. ▪ 18 GSI indexes across 10 buckets
-
Untriaged
-
Unknown
Description
1. load data+indexes
2. rebalances in, out of data etc — possibly not related w this issue
3. Keep mutations on the cluster overnight.
See indexer crashes as below
Service 'indexer' exited with status 1. Restarting. Messages: 2015-07-24T02:07:13.167Z-07:00 [Info] StorageMgr::handleCreateSnapshot
Added New Snapshot Index: 13031476919029327009 PartitionId: 0 SliceId: 0 Crc64: 7547764960941221267 (SnapshotInfo: seqnos: 50, 50, 0 committed:false)
2015-07-24T02:07:13.213Z-07:00 [Info] StorageMgr::handleCreateSnapshot
Added New Snapshot Index: 10252050856808983086 PartitionId: 0 SliceId: 0 Crc64: 16904070808310454387 (SnapshotInfo: seqnos: 79359, 79359, 6195 committed:true)
[goport] 2015/07/24 02:07:22 /opt/couchbase/bin/indexer terminated: signal: killed
The indexer.2.log shows nothing unusual or no ooms on other logs.
2015-07-24T02:07:13.213Z-07:00 [Info] StorageMgr::handleCreateSnapshot
|
Added New Snapshot Index: 10252050856808983086 PartitionId: 0 SliceId: 0 Crc64: 16904070808310454387 (SnapshotInfo: seqnos: 79359, 79359, 6195 committed:true)
|
[goport] 2015/07/24 02:07:22 /opt/couchbase/bin/indexer terminated: signal: killed
|
2015-07-24T02:07:23.06Z-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/disk0/index/@2i]
|
attached logs from index node. 183
adding logs from other nodes in the system.
