Details
Description
1. Load 20M documents into a cluster with an indexer (FDB as the index memory) and compact the bucket
2. Build an index from the loaded document.
3. Wait for the index completion.
4. Start 20M mutations on the bucket (100% updates)
5. Wait for the index to finish the incremental build
During the incremental build phase, the indexer is blocked by FDB get_kv call. This call is struck for really long time. Following is from the logs analyzed.
1 @ 0x4065ab 0x4afcd1 0x4c0258 0x4db05a 0x4d8a1b 0x4d84e6 0x464271
- 0x4afcd1 github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_set_kv+0x41 ??:0
- 0x4c0258 github.com/couchbase/indexing/secondary/fdb.(*KVStore).SetKV+0x208 /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/kv.go:65
- 0x4db05a github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insertSecIndex+0x171a /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:464
- 0x4d8a1b github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insert+0xeb /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:353
- 0x4d84e6 github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).handleCommandsWorker+0x4c6 /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:314
The forestDB files for the indexer can be found at /home/fdbfiles in 172.23.99.247 (root:couchbase)