Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
Triaged
-
Unknown
-
Storage-Sprint-End-Aug-10-2018
Description
The indexer is stuck in warmup state from 2018-07-30T08:14:40.221+02:00 (i.e. from it was last restarted), till 2018-07-30T08:51:12.819+02:00 (when logs were collected). i.e. Indexer was stuck in warmup for ~ 35 mins.
As per the grouting dump and the indexer logs, we can see than there are 10 indexes, out of which 9 have been recovered from storage. But the 10th index (InstanceId: 14621354403389861515 Name: config_quer Bucket: OBG-sportsbook), is waiting for initStores to complete.
1 @ 0x4348ca 0x4349be 0x444b3d 0x4447d0 0x6bee47 0x577e4f 0x570a4e 0x56fd0b 0x56d3a2 0x56c55c 0x53b29d 0x523c94 0x52f9da 0x634ef3 0x465a11 |
# 0x4447cf sync.runtime_Semacquire+0x2f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/sema.go:47 |
# 0x6bee46 sync.(*WaitGroup).Wait+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/sync/waitgroup.go:131 |
# 0x577e4e github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).restore+0x13e goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1244 |
# 0x570a4d github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).doRecovery+0x1ad goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 |
# 0x56fd0a github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores+0x223a goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:360 |
# 0x56d3a1 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice+0xd71 goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:198 |
# 0x56c55b github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice+0xeb goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:27 |
# 0x53b29c github.com/couchbase/indexing/secondary/indexer.NewSlice+0x6ec goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6049 |
# 0x523c93 github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance+0x323 goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:3508 |
# 0x52f9d9 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState+0x2a9 goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4905 |
# 0x634ef2 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1+0x42 goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4607 |
Waiting for lssCleanerMutex.
1 @ 0x4348ca 0x4349be 0x444b3d 0x4447d0 0x6bd940 0xa80b45 0x63dadd 0x465a11 |
# 0x4447cf sync.runtime_Semacquire+0x2f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/sema.go:47 |
# 0x6bd93f sync.(*Mutex).Lock+0xcf /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/sync/mutex.go:85 |
# 0xa80b44 github.com/couchbase/plasma.(*Plasma).Rollback+0x74 goproj/src/github.com/couchbase/plasma/mvcc.go:481 |
# 0x63dadc github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).restore.func1+0x6c goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1227 |
|
lss cleaner seems to be actually in progress.
1 @ 0x6bd4d5 0x6b9f2a 0x46e3ae 0xa9f260 0xa7810e 0xa79c5a 0xaa18b4 0xa7c19d 0xa7b333 0xa81c10 0xaa2667 0xaa193d 0xa7b20e 0xa7af32 0xa7da85 0xa7e2b9 0x465a11 |
# 0x6bd4d4 syscall.Syscall+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/syscall/asm_linux_amd64.s |
:18 |
# 0x6b9f29 syscall.Fsync+0x49 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/syscall/zsyscall_linux_am |
d64.go:492 |
# 0x46e3ad os.(*File).Sync+0x3d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/os/file_posix.go:121 |
# 0xa9f25f github.com/couchbase/plasma.syncFile+0x3f goproj/src/github.com/couchbase/plasma/util.go:256 |
# 0xa7810d github.com/couchbase/plasma.(*multiFilelog).Commit+0x11d goproj/src/github.com/couchbase/plasma/log.go:398 |
# 0xa79c59 github.com/couchbase/plasma.(*lsStore).flush+0x249 goproj/src/github.com/couchbase/plasma/lss.go:205 |
# 0xaa18b3 github.com/couchbase/plasma.(*lsStore).(github.com/couchbase/plasma.flush)-fm+0x33 goproj/src/github.com/couchbase/plasma/lss.go:147 |
# 0xa7c19c github.com/couchbase/plasma.(*flushBuffer).Done+0xdc goproj/src/github.com/couchbase/plasma/lss.go:708 |
# 0xa7b332 github.com/couchbase/plasma.(*lsStore).Sync+0x92 goproj/src/github.com/couchbase/plasma/lss.go:472 |
# 0xa81c0f github.com/couchbase/plasma.(*Plasma).updateMaxSn+0x1af goproj/src/github.com/couchbase/plasma/mvcc.go:620 |
# 0xaa2666 github.com/couchbase/plasma.(*Plasma).CleanLSS.func1+0xba6 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:313 |
# 0xaa193c github.com/couchbase/plasma.(*lsStore).RunCleaner.func1+0x6c goproj/src/github.com/couchbase/plasma/lss.go:407 |
# 0xa7b20d github.com/couchbase/plasma.(*lsStore).Visitor2+0x16d goproj/src/github.com/couchbase/plasma/lss.go:442 |
# 0xa7af31 github.com/couchbase/plasma.(*lsStore).RunCleaner+0x141 goproj/src/github.com/couchbase/plasma/lss.go:421 |
# 0xa7da84 github.com/couchbase/plasma.(*Plasma).CleanLSS+0x464 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:330 |
# 0xa7e2b8 github.com/couchbase/plasma.(*Plasma).lssCleanerDaemon+0x178 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:377 |
1 @ 0x6bd545 0x6baff7 0x46f929 0x46d5ac 0xa7789c 0xa79a8e 0xaa18b4 0xa7c19d 0xa7b333 0xa81c10 0xaa2667 0xaa193d 0xa7b20e 0xa7af32 0xa7da85 0xa7e2b9 0x465a11 |
# 0x6bd544 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/syscall/asm_linux_amd64.s:44 |
# 0x6baff6 syscall.Pwrite+0x76 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/syscall/zsyscall_linux_amd64.go:1389 |
# 0x46f928 os.(*File).pwrite+0x58 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/os/file_unix.go:275 |
# 0x46d5ab os.(*File).WriteAt+0x9b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/os/file.go:166 |
# 0xa7789b github.com/couchbase/plasma.(*multiFilelog).Append+0x14b goproj/src/github.com/couchbase/plasma/log.go:316 |
# 0xa79a8d github.com/couchbase/plasma.(*lsStore).flush+0x7d goproj/src/github.com/couchbase/plasma/lss.go:186 |
# 0xaa18b3 github.com/couchbase/plasma.(*lsStore).(github.com/couchbase/plasma.flush)-fm+0x33 goproj/src/github.com/couchbase/plasma/lss.go:147 |
# 0xa7c19c github.com/couchbase/plasma.(*flushBuffer).Done+0xdc goproj/src/github.com/couchbase/plasma/lss.go:708 |
# 0xa7b332 github.com/couchbase/plasma.(*lsStore).Sync+0x92 goproj/src/github.com/couchbase/plasma/lss.go:472 |
# 0xa81c0f github.com/couchbase/plasma.(*Plasma).updateMaxSn+0x1af goproj/src/github.com/couchbase/plasma/mvcc.go:620 |
# 0xaa2666 github.com/couchbase/plasma.(*Plasma).CleanLSS.func1+0xba6 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:313 |
# 0xaa193c github.com/couchbase/plasma.(*lsStore).RunCleaner.func1+0x6c goproj/src/github.com/couchbase/plasma/lss.go:407 |
# 0xa7b20d github.com/couchbase/plasma.(*lsStore).Visitor2+0x16d goproj/src/github.com/couchbase/plasma/lss.go:442 |
# 0xa7af31 github.com/couchbase/plasma.(*lsStore).RunCleaner+0x141 goproj/src/github.com/couchbase/plasma/lss.go:421 |
# 0xa7da84 github.com/couchbase/plasma.(*Plasma).CleanLSS+0x464 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:330 |
# 0xa7e2b8 github.com/couchbase/plasma.(*Plasma).lssCleanerDaemon+0x178 goproj/src/github.com/couchbase/plasma/lss_cleaner.go:377 |
