Details
-
Bug
-
Status: Closed
-
Test Blocker
-
Resolution: Fixed
-
1.0.0
Description
Operator gets stalled while performing resize related testing on openshift platform running on AWS.
As per the operator logs, operator logs started from 2018-08-10T03:23:01Z to 2018-08-10T05:32:02Z
But operator pod is running more than 3hrs as per kube data.
[ec2-user@ip-172-16-0-187 ~]$ kubectl get pods
|
NAME READY STATUS RESTARTS AGE
|
couchbase-operator-554b978699-tl49x 1/1 Running 0 3h
|
docker-registry-1-9fznr 1/1 Running 1 2d
|
registry-console-1-jcn85 1/1 Running 1 2d
|
router-1-mkhbw 1/1 Running 1 2d
|
test-couchbase-gcjn8-0000 1/1 Running 0 54m
|
|
[ec2-user@ip-172-16-0-187 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-16-0-134.us-west-2.compute.internal Ready compute 7d v1.9.1+a0ce1bc657
ip-172-16-0-187.us-west-2.compute.internal Ready master 7d v1.9.1+a0ce1bc657
ip-172-16-0-229.us-west-2.compute.internal Ready compute 7d v1.9.1+a0ce1bc657
ip-172-16-0-239.us-west-2.compute.internal Ready compute 7d v1.9.1+a0ce1bc657
[ec2-user@ip-172-16-0-187 ~]${noformat}
Attachments
Activity
Opened K8S-527 to perform this step automatically in the 1.1.0 release
I am getting Permission denied error code
ec2-user@ip-172-16-0-187 ~]$ kubectl exec couchbase-operator-554b978699-tl49x wget localhost:8080/debug/pprof/goroutine?debug=1
|
Connecting to localhost:8080 (127.0.0.1:8080)
|
wget: can't open 'goroutine?debug=1': Permission denied
|
command terminated with exit code 1
|
That's a file system write error. You may also wish to look at the /v1/stats/cluster end point too as that contains runloop stats
/tmp $ cat cluster
|
{"clusters":{"test-couchbase-gcjn8":{"last_reconcile_loop_start_time":"2018-08-10T09:56:50.534193208Z","last_reconcile_loop_end_time":"2018-08-10T09:56:50.579002946Z","last_completed_loop_duration_sec":0.044809768,"reconcile_loop_status":"sleeping","reconcile_loop_sleep_time":8,"clusterPhase":"Running","controlPaused":false}}}
|
|
/tmp $ cat goroutine?debug\=1
|
goroutine profile: total 200
|
105 @ 0x42d69c 0x42d78e 0x403d2b 0x403ab3 0x8c463e 0x45c191
|
# 0x8c463d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:114
|
|
37 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xfc2ebc 0x45c191
|
# 0xfc2ebb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0xab /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record/event.go:229
|
|
36 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c3d51 0x45c191
|
# 0x8c3d50 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x60 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208
|
|
2 @ 0x42d69c 0x43d9f8 0x8c3127 0x45c191
|
# 0x8c3126 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1+0x196 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354
|
|
1 @ 0x42d69c 0x40502b 0x4047fb 0xff4554 0x45c191
|
# 0xff4553 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:103
|
|
1 @ 0x42d69c 0x40502b 0x4047fb 0xffd8af 0x45c191
|
# 0xffd8ae github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).Run+0x2ee /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:90
|
|
1 @ 0x42d69c 0x40502b 0x4047fb 0xfff000 0xfffd34 0x100675f 0x45c191
|
# 0xffefff github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk.Run+0xbf /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/api.go:69
|
# 0xfffd33 github.com/couchbase/couchbase-operator/pkg/controller.(*Controller).Start+0x303 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:95
|
# 0x100675e main.run+0x12e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:136
|
|
1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x5d0b45 0x5d10b0 0x5d4c00 0x53265b 0x4736c6 0x473838 0x8a213b 0x8a2aa4 0x8b40d0 0x8b38fd 0x45c191
|
# 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173
|
# 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
|
# 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
|
# 0x497259 internal/poll.(*FD).Read+0x189 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125
|
# 0x57d351 net.(*netFD).Read+0x51 /usr/local/go/1.9/go/src/net/fd_unix.go:202
|
# 0x58f8ac net.(*conn).Read+0x6c /usr/local/go/1.9/go/src/net/net.go:176
|
# 0x5d0b44 crypto/tls.(*block).readFromUntil+0x94 /usr/local/go/1.9/go/src/crypto/tls/conn.go:488
|
# 0x5d10af crypto/tls.(*Conn).readRecord+0xdf /usr/local/go/1.9/go/src/crypto/tls/conn.go:590
|
# 0x5d4bff crypto/tls.(*Conn).Read+0x10f /usr/local/go/1.9/go/src/crypto/tls/conn.go:1134
|
# 0x53265a bufio.(*Reader).Read+0x30a /usr/local/go/1.9/go/src/bufio/bufio.go:213
|
# 0x4736c5 io.ReadAtLeast+0x85 /usr/local/go/1.9/go/src/io/io.go:309
|
# 0x473837 io.ReadFull+0x57 /usr/local/go/1.9/go/src/io/io.go:327
|
# 0x8a213a github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.readFrameHeader+0x7a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:237
|
# 0x8a2aa3 github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:492
|
# 0x8b40cf github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run+0x8f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1428
|
# 0x8b38fc github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop+0x9c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1354
|
|
1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x690d10 0x531e8a 0x531fda 0x6917e5 0x45c191
|
# 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173
|
# 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
|
# 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
|
# 0x497259 internal/poll.(*FD).Read+0x189 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125
|
# 0x57d351 net.(*netFD).Read+0x51 /usr/local/go/1.9/go/src/net/fd_unix.go:202
|
# 0x58f8ac net.(*conn).Read+0x6c /usr/local/go/1.9/go/src/net/net.go:176
|
# 0x690d0f net/http.(*persistConn).Read+0x13f /usr/local/go/1.9/go/src/net/http/transport.go:1391
|
# 0x531e89 bufio.(*Reader).fill+0x119 /usr/local/go/1.9/go/src/bufio/bufio.go:97
|
# 0x531fd9 bufio.(*Reader).Peek+0x39 /usr/local/go/1.9/go/src/bufio/bufio.go:129
|
# 0x6917e4 net/http.(*persistConn).readLoop+0x184 /usr/local/go/1.9/go/src/net/http/transport.go:1539
|
|
1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x498912 0x57dd72 0x599aee 0x597ec9 0x682cdf 0x681982 0x6816c9 0x68285f 0x45c191
|
# 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173
|
# 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
|
# 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
|
# 0x498911 internal/poll.(*FD).Accept+0x1e1 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:334
|
# 0x57dd71 net.(*netFD).accept+0x41 /usr/local/go/1.9/go/src/net/fd_unix.go:238
|
# 0x599aed net.(*TCPListener).accept+0x2d /usr/local/go/1.9/go/src/net/tcpsock_posix.go:136
|
# 0x597ec8 net.(*TCPListener).AcceptTCP+0x48 /usr/local/go/1.9/go/src/net/tcpsock.go:234
|
# 0x682cde net/http.tcpKeepAliveListener.Accept+0x2e /usr/local/go/1.9/go/src/net/http/server.go:3120
|
# 0x681981 net/http.(*Server).Serve+0x1b1 /usr/local/go/1.9/go/src/net/http/server.go:2695
|
# 0x6816c8 net/http.(*Server).ListenAndServe+0xa8 /usr/local/go/1.9/go/src/net/http/server.go:2636
|
# 0x68285e net/http.ListenAndServe+0x7e /usr/local/go/1.9/go/src/net/http/server.go:2882
|
|
1 @ 0x42d69c 0x42d78e 0x404b54 0x4047fb 0xfefbf6 0xff54e4 0x8c2e11 0x8c2ecf 0x45c191
|
# 0xfefbf5 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x45 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:427
|
# 0xff54e3 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:219
|
# 0x8c2e10 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
|
# 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
|
|
1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x860c2f 0x45c191
|
# 0x860c2e github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon+0x9e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog/glog.go:879
|
|
1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c2c18 0x8c2af4 0x8c2a2d 0x10057d8 0x8c2f5e 0x8c27cd 0x8c26fd 0x1004972 0x10046a0 0x100473a 0x10064a4 0x42d1fd 0x45c191
|
# 0x8c2c17 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor+0x57 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:310
|
# 0x8c2af3 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal+0x93 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:227
|
# 0x8c2a2c github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Poll+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:221
|
# 0x10057d7 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1+0x97 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:193
|
# 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
|
# 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
|
# 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
|
# 0x1004971 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x71 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:192
|
# 0x100469f github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x9f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:146
|
# 0x1004739 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie+0x59 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:157
|
# 0x10064a3 main.main+0x6a3 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:115
|
# 0x42d1fc runtime.main+0x20c /usr/local/go/1.9/go/src/runtime/proc.go:185
|
|
1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xff0725 0xff552a 0x8c2ecf 0x45c191
|
# 0xff0724 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545
|
# 0xff5529 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:387
|
# 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
|
|
1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0x8aac9c 0x8b6141 0x6e6234 0x6e5f25 0x6e58c8 0xa1c178 0xe78795 0xe78e4c 0x8c4658 0x45c191
|
# 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507
|
# 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56
|
# 0x8aac9b github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*pipe).Read+0x9b /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/pipe.go:64
|
# 0x8b6140 github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read+0xb0 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1674
|
# 0x6e6233 encoding/json.(*Decoder).refill+0x113 /usr/local/go/1.9/go/src/encoding/json/stream.go:152
|
# 0x6e5f24 encoding/json.(*Decoder).readValue+0x274 /usr/local/go/1.9/go/src/encoding/json/stream.go:128
|
# 0x6e58c7 encoding/json.(*Decoder).Decode+0x77 /usr/local/go/1.9/go/src/encoding/json/stream.go:57
|
# 0xa1c177 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x2c7 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150
|
# 0xe78794 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77
|
# 0xe78e4b github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x7b /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49
|
# 0x8c4657 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93
|
|
1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xfe7736 0xfe51c0 0xff533a 0x8c2f5e 0x8c27cd 0x8c26fd 0xfe4fe7 0xfedf9a 0x45c191
|
# 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507
|
# 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56
|
# 0xfe7735 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0xb5 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438
|
# 0xfe51bf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:150
|
# 0xff5339 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124
|
# 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
|
# 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
|
# 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
|
# 0xfe4fe6 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x2b6 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124
|
# 0xfedf99 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x4c9 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:226
|
|
1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xffac75 0xffb6bc 0xffbbec 0xffdddd 0xffdd7b 0xffe92a 0x8c2f5e 0x8c27cd 0x8c26fd 0x45c191
|
# 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507
|
# 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56
|
# 0xffac74 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*Type).Get+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:124
|
# 0xffb6bb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get+0x3b <autogenerated>:1
|
# 0xffbbeb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get+0x3b <autogenerated>:1
|
# 0xffdddc github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).processNextItem+0x3c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:43
|
# 0xffdd7a github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).runWorker+0x2a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:37
|
# 0xffe929 github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).(github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.runWorker)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:88
|
# 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
|
# 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
|
# 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
|
|
1 @ 0x42d69c 0x43d9f8 0x692f95 0x45c191
|
# 0x692f94 net/http.(*persistConn).writeLoop+0x164 /usr/local/go/1.9/go/src/net/http/transport.go:1759
|
|
1 @ 0x42d69c 0x43d9f8 0xfc61b6 0xfda64b 0x45c191
|
# 0xfc61b5 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run+0x235 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:294
|
# 0xfda64a github.com/couchbase/couchbase-operator/pkg/cluster.New.func1+0x27a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150
|
|
1 @ 0x42d69c 0x43d9f8 0xfebcd8 0xfeb64c 0xff45b3 0x8c2f5e 0x8c27cd 0x8c26fd 0xfea514 0xff52f4 0x8c2e11 0x8c2ecf 0x45c191
|
# 0xfebcd7 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x287 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:373
|
# 0xfeb64b github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0xffb /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:339
|
# 0xff45b2 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x32 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:204
|
# 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
|
# 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
|
# 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
|
# 0xfea513 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run+0x163 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:203
|
# 0xff52f3 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:122
|
# 0x8c2e10 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
|
# 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
|
|
1 @ 0x42d69c 0x43d9f8 0xff04d0 0xff556a 0x8c2ecf 0x45c191
|
# 0xff04cf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop+0x18f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:519
|
# 0xff5569 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.pop)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388
|
# 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
|
|
1 @ 0x42d69c 0x43d9f8 0xff47d0 0x45c191
|
# 0xff47cf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1+0x18f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:278
|
|
1 @ 0x42d69c 0x43d9f8 0xff95ef 0x45c191
|
# 0xff95ee github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x39e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:208
|
|
1 @ 0x7071b2 0x706fb0 0x703b0b 0x70fc48 0x70ff4b 0x67e9d4 0x680080 0x6815f4 0x67d8ad 0x45c191
|
# 0x7071b1 runtime/pprof.writeRuntimeProfile+0xa1 /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:637
|
# 0x706faf runtime/pprof.writeGoroutine+0x9f /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:599
|
# 0x703b0a runtime/pprof.(*Profile).WriteTo+0x3aa /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:310
|
# 0x70fc47 net/http/pprof.handler.ServeHTTP+0x1b7 /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:237
|
# 0x70ff4a net/http/pprof.Index+0x1da /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:248
|
# 0x67e9d3 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/1.9/go/src/net/http/server.go:1918
|
# 0x68007f net/http.(*ServeMux).ServeHTTP+0x12f /usr/local/go/1.9/go/src/net/http/server.go:2254
|
# 0x6815f3 net/http.serverHandler.ServeHTTP+0xb3 /usr/local/go/1.9/go/src/net/http/server.go:2619
|
# 0x67d8ac net/http.(*conn).serve+0x71c /usr/local/go/1.9/go/src/net/http/server.go:1801
|
Okay so it's 11:21+0100 now, which means in UTC that's 10:21. You added that comment 21 minutes ago, so 10:00 and the time stamp says the last reconcile was 09:56. Do you agree it is working and not hanging?
Yeah. I too agree that there is no problem with the operator now.
But there is something wrong in the AWS node on which the testing is running.
I have restarted the test now. Will keep you posted on further updates.
Not seeing this problem now. Looks like there was some runtime related issues.
After rebooting the target node, tests seems to work fine.
Not enough info to debug. Can you exec into the operator pod and run:
wget localhost:8080/debug/pprof/goroutine?debug=1
then post the output.