Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 4.5.0
Affects Version/s: 4.0.0, 4.1.0
Component/s: forestdb, secondary-index
Security Level: Public
Labels:
None
Environment:

Hide
[info] OS Name : Linux 3.18.6-1.el7.elrepo.x86_64
[info] OS Version : Red Hat Enterprise Linux Server release 7.1 (Maipo)
[warn] CB Version : 4.1.0-4958-enterprise - Non-default build
[info] CB Packages : couchbase-server-4.1.0-4958.x86_64

Show
[info] OS Name : Linux 3.18.6-1.el7.elrepo.x86_64 [info] OS Version : Red Hat Enterprise Linux Server release 7.1 (Maipo) [warn] CB Version : 4.1.0-4958-enterprise - Non-default build [info] CB Packages : couchbase-server-4.1.0-4958.x86_64

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
cbcollect logs from stuck 3 Nodes:-
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.169.3.221.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.164.145.124.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.231.109.zip

cbcollect logs from Remaining Nodes:-

https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.164.20.11.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.28.233.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.166.141.123.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.1.250.zip
https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.147.140.62.zip

GDB stacktrace from stuck 3 nodes
https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-124.out
https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-221.out
https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-109.out

pprof output from stuck 3 nodes

curl <ip>:9102/debug/pprof/goroutine?debug=1 -u Administrator:<password>

https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.169.3.221.out2
https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.164.145.124.out2
https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.168.231.109.out2

Show
cbcollect logs from stuck 3 Nodes:- https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.169.3.221.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.164.145.124.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.231.109.zip cbcollect logs from Remaining Nodes:- https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.164.20.11.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.28.233.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.166.141.123.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.168.1.250.zip https://s3.amazonaws.com/cb-customers/bastian/832/collectinfo-2015-12-08T093818-ns_1%4010.147.140.62.zip GDB stacktrace from stuck 3 nodes https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-124.out https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-221.out https://s3.amazonaws.com/customers.couchbase.com/bastian/gdb-109.out pprof output from stuck 3 nodes curl <ip>:9102/debug/pprof/goroutine?debug=1 -u Administrator:<password> https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.169.3.221.out2 https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.164.145.124.out2 https://s3.amazonaws.com/customers.couchbase.com/bastian/pprof-10.168.231.109.out2
Is this a Regression?:
Unknown

Description

Index build is getting hang in Forestdb. The goroutine dump shows that compaction thread and a forestdb getkv call are stuck.

2 @ 0x4065e5 0x480c73 0x48ba44 0x4a3aee 0x4a0d60 0x49ffbc 0x49fb70 0x42d8e1
0x480c73 github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_get_kv+ 0x43 /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/ couchbase/indexing/secondary/fdb/:172

1 @ 0x4065e5 0x480693 0x484b5b 0x4a7b42 0x54b013 0x42d8e1
0x480693 github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_compact_upto+0x43 /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/ indexing/secondary/fdb/:64

The goroutine dump can be captured via:

curl <ip>:9102/debug/pprof/goroutine?debug=1 -u Administrator:<password> > pprof.out