Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.0.0
-
Security Level: Public
-
400-3502
system test setup.
debug log on for indexer.
▪ Machine configuration:
⁃ SSDs
⁃ 24GB Ram
⁃ 250GB /data and /index disks
⁃ 8 Core CPU
⁃ Individual VMs ( non-shared)
▪ Indexer Settings : ( see attached settings.json)
⁃ Index Ram Quota on Index Nodes : 21GB,
⁃ 8 Indexer Threads
⁃ indexer.settings.scan_timeout:1400000 No index settings changed at any point on the test.
▪ 8 Node cluster
⁃ 2 Index nodes,
⁃ 2 Query nodes,
⁃ 4 KV nodes
▪ 10 Buckets
▪ 50 warehouses for tppc(9) buckets and 5M items for default.
▪ 18 GSI indexes across 10 buckets
400-3502 system test setup. debug log on for indexer. ▪ Machine configuration: ⁃ SSDs ⁃ 24GB Ram ⁃ 250GB /data and /index disks ⁃ 8 Core CPU ⁃ Individual VMs ( non-shared) ▪ Indexer Settings : ( see attached settings.json) ⁃ Index Ram Quota on Index Nodes : 21GB, ⁃ 8 Indexer Threads ⁃ indexer.settings.scan_timeout:1400000 No index settings changed at any point on the test. ▪ 8 Node cluster ⁃ 2 Index nodes, ⁃ 2 Query nodes, ⁃ 4 KV nodes ▪ 10 Buckets ▪ 50 warehouses for tppc(9) buckets and 5M items for default. ▪ 18 GSI indexes across 10 buckets
-
Untriaged
-
Unknown
Description
1. load data + indexes - success
2. mutations for 2hours
3. rebalance out Index Node, 183
4. Rebalance In Index Node, 183
5. Recreate indexes on 183 node — seeing a number of indexes are not getting built, they appear to be stuck
2015-07-23T11:52:03.814Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 281970 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.844Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282000 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.874Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282030 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.904Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282060 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.934Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282090 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.964Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282120 Milliseconds Vbucket 512
|
2015-07-23T11:52:03.994Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282150 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.024Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282180 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.054Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282210 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.084Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282240 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.114Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282270 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.144Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282300 Milliseconds Vbucket 512
|
2015-07-23T11:52:04.174Z-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 282330 Milliseconds Vbucket 512
|
For this drop index, fails w/ GSI request time out.
kill -9 indexer process, appears to restore the indexes and I can build.
Can reproduce on the same setup, twice so far.
