[K8S-524] Operator get stalled when performing resize operations in couchbase cluster - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Test Blocker
Resolution: Fixed
Affects Version/s: 1.0.0
Fix Version/s: 1.0.0
Component/s: openshift, operator
Labels:
- functional-test

Description

Operator gets stalled while performing resize related testing on openshift platform running on AWS.

As per the operator logs, operator logs started from 2018-08-10T03:23:01Z to 2018-08-10T05:32:02Z

But operator pod is running more than 3hrs as per kube data.

[ec2-user@ip-172-16-0-187 ~]$ kubectl get pods

NAME                                  READY     STATUS    RESTARTS   AGE

couchbase-operator-554b978699-tl49x   1/1       Running   0          3h

docker-registry-1-9fznr               1/1       Running   1          2d

registry-console-1-jcn85              1/1       Running   1          2d

router-1-mkhbw                        1/1       Running   1          2d

test-couchbase-gcjn8-0000             1/1       Running   0          54m

[ec2-user@ip-172-16-0-187 ~]$ kubectl get nodes
NAME                                         STATUS    ROLES     AGE       VERSION
ip-172-16-0-134.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
ip-172-16-0-187.us-west-2.compute.internal   Ready     master    7d        v1.9.1+a0ce1bc657
ip-172-16-0-229.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
ip-172-16-0-239.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
[ec2-user@ip-172-16-0-187 ~]${noformat}

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

cbopinfo-20180810T061229+0000.tar.gz
60 kB
09/Aug/18 11:20 PM

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Simon Murray added a comment - 10/Aug/18 1:31 AM

Not enough info to debug. Can you exec into the operator pod and run:

wget localhost:8080/debug/pprof/goroutine?debug=1

then post the output.

Simon Murray added a comment - 10/Aug/18 1:31 AM Not enough info to debug. Can you exec into the operator pod and run: wget localhost:8080/debug/pprof/goroutine?debug=1 then post the output.

Simon Murray added a comment - 10/Aug/18 1:35 AM

Opened ~~K8S-527~~ to perform this step automatically in the 1.1.0 release

Simon Murray added a comment - 10/Aug/18 1:35 AM Opened K8S-527 to perform this step automatically in the 1.1.0 release

Ashwin Govindarajulu added a comment - 10/Aug/18 1:48 AM

I am getting Permission denied error code

ec2-user@ip-172-16-0-187 ~]$ kubectl exec couchbase-operator-554b978699-tl49x wget localhost:8080/debug/pprof/goroutine?debug=1

Connecting to localhost:8080 (127.0.0.1:8080)

wget: can't open 'goroutine?debug=1': Permission denied

command terminated with exit code 1

Ashwin Govindarajulu added a comment - 10/Aug/18 1:48 AM I am getting Permission denied error code ec2-user@ip-172-16-0-187 ~]$ kubectl exec couchbase-operator-554b978699-tl49x wget localhost:8080/debug/pprof/goroutine?debug=1 Connecting to localhost:8080 (127.0.0.1:8080) wget: can't open 'goroutine?debug=1': Permission denied command terminated with exit code 1

Simon Murray added a comment - 10/Aug/18 2:53 AM

That's a file system write error. You may also wish to look at the /v1/stats/cluster end point too as that contains runloop stats

Simon Murray added a comment - 10/Aug/18 2:53 AM That's a file system write error. You may also wish to look at the /v1/stats/cluster end point too as that contains runloop stats

Ashwin Govindarajulu added a comment - 10/Aug/18 3:00 AM

/tmp $ cat cluster

{"clusters":{"test-couchbase-gcjn8":{"last_reconcile_loop_start_time":"2018-08-10T09:56:50.534193208Z","last_reconcile_loop_end_time":"2018-08-10T09:56:50.579002946Z","last_completed_loop_duration_sec":0.044809768,"reconcile_loop_status":"sleeping","reconcile_loop_sleep_time":8,"clusterPhase":"Running","controlPaused":false}}}

/tmp $ cat goroutine?debug\=1

goroutine profile: total 200

105 @ 0x42d69c 0x42d78e 0x403d2b 0x403ab3 0x8c463e 0x45c191

#    0x8c463d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11d    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:114

37 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xfc2ebc 0x45c191

#    0xfc2ebb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0xab    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record/event.go:229

36 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c3d51 0x45c191

#    0x8c3d50    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x60    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208

2 @ 0x42d69c 0x43d9f8 0x8c3127 0x45c191

#    0x8c3126    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1+0x196    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354

1 @ 0x42d69c 0x40502b 0x4047fb 0xff4554 0x45c191

#    0xff4553    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:103

1 @ 0x42d69c 0x40502b 0x4047fb 0xffd8af 0x45c191

#    0xffd8ae    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).Run+0x2ee    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:90

1 @ 0x42d69c 0x40502b 0x4047fb 0xfff000 0xfffd34 0x100675f 0x45c191

#    0xffefff    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk.Run+0xbf    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/api.go:69

#    0xfffd33    github.com/couchbase/couchbase-operator/pkg/controller.(*Controller).Start+0x303        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:95

#    0x100675e    main.run+0x12e                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:136

1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x5d0b45 0x5d10b0 0x5d4c00 0x53265b 0x4736c6 0x473838 0x8a213b 0x8a2aa4 0x8b40d0 0x8b38fd 0x45c191

#    0x427dd6    internal/poll.runtime_pollWait+0x56                                    /usr/local/go/1.9/go/src/runtime/netpoll.go:173

#    0x49643d    internal/poll.(*pollDesc).wait+0xad                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85

#    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90

#    0x497259    internal/poll.(*FD).Read+0x189                                        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125

#    0x57d351    net.(*netFD).Read+0x51                                            /usr/local/go/1.9/go/src/net/fd_unix.go:202

#    0x58f8ac    net.(*conn).Read+0x6c                                            /usr/local/go/1.9/go/src/net/net.go:176

#    0x5d0b44    crypto/tls.(*block).readFromUntil+0x94                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:488

#    0x5d10af    crypto/tls.(*Conn).readRecord+0xdf                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:590

#    0x5d4bff    crypto/tls.(*Conn).Read+0x10f                                        /usr/local/go/1.9/go/src/crypto/tls/conn.go:1134

#    0x53265a    bufio.(*Reader).Read+0x30a                                        /usr/local/go/1.9/go/src/bufio/bufio.go:213

#    0x4736c5    io.ReadAtLeast+0x85                                            /usr/local/go/1.9/go/src/io/io.go:309

#    0x473837    io.ReadFull+0x57                                            /usr/local/go/1.9/go/src/io/io.go:327

#    0x8a213a    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.readFrameHeader+0x7a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:237

#    0x8a2aa3    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:492

#    0x8b40cf    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run+0x8f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1428

#    0x8b38fc    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop+0x9c    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1354

1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x690d10 0x531e8a 0x531fda 0x6917e5 0x45c191

#    0x427dd6    internal/poll.runtime_pollWait+0x56    /usr/local/go/1.9/go/src/runtime/netpoll.go:173

#    0x49643d    internal/poll.(*pollDesc).wait+0xad    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85

#    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90

#    0x497259    internal/poll.(*FD).Read+0x189        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125

#    0x57d351    net.(*netFD).Read+0x51            /usr/local/go/1.9/go/src/net/fd_unix.go:202

#    0x58f8ac    net.(*conn).Read+0x6c            /usr/local/go/1.9/go/src/net/net.go:176

#    0x690d0f    net/http.(*persistConn).Read+0x13f    /usr/local/go/1.9/go/src/net/http/transport.go:1391

#    0x531e89    bufio.(*Reader).fill+0x119        /usr/local/go/1.9/go/src/bufio/bufio.go:97

#    0x531fd9    bufio.(*Reader).Peek+0x39        /usr/local/go/1.9/go/src/bufio/bufio.go:129

#    0x6917e4    net/http.(*persistConn).readLoop+0x184    /usr/local/go/1.9/go/src/net/http/transport.go:1539

1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x498912 0x57dd72 0x599aee 0x597ec9 0x682cdf 0x681982 0x6816c9 0x68285f 0x45c191

#    0x427dd6    internal/poll.runtime_pollWait+0x56        /usr/local/go/1.9/go/src/runtime/netpoll.go:173

#    0x49643d    internal/poll.(*pollDesc).wait+0xad        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85

#    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90

#    0x498911    internal/poll.(*FD).Accept+0x1e1        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:334

#    0x57dd71    net.(*netFD).accept+0x41            /usr/local/go/1.9/go/src/net/fd_unix.go:238

#    0x599aed    net.(*TCPListener).accept+0x2d            /usr/local/go/1.9/go/src/net/tcpsock_posix.go:136

#    0x597ec8    net.(*TCPListener).AcceptTCP+0x48        /usr/local/go/1.9/go/src/net/tcpsock.go:234

#    0x682cde    net/http.tcpKeepAliveListener.Accept+0x2e    /usr/local/go/1.9/go/src/net/http/server.go:3120

#    0x681981    net/http.(*Server).Serve+0x1b1            /usr/local/go/1.9/go/src/net/http/server.go:2695

#    0x6816c8    net/http.(*Server).ListenAndServe+0xa8        /usr/local/go/1.9/go/src/net/http/server.go:2636

#    0x68285e    net/http.ListenAndServe+0x7e            /usr/local/go/1.9/go/src/net/http/server.go:2882

1 @ 0x42d69c 0x42d78e 0x404b54 0x4047fb 0xfefbf6 0xff54e4 0x8c2e11 0x8c2ecf 0x45c191

#    0xfefbf5    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x45                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:427

#    0xff54e3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:219

#    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30                                    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54

#    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71

1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x860c2f 0x45c191

#    0x860c2e    github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon+0x9e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog/glog.go:879

1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c2c18 0x8c2af4 0x8c2a2d 0x10057d8 0x8c2f5e 0x8c27cd 0x8c26fd 0x1004972 0x10046a0 0x100473a 0x10064a4 0x42d1fd 0x45c191

#    0x8c2c17    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor+0x57                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:310

#    0x8c2af3    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal+0x93            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:227

#    0x8c2a2c    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Poll+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:221

#    0x10057d7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1+0x97    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:193

#    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133

#    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134

#    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

#    0x1004971    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x71    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:192

#    0x100469f    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x9f        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:146

#    0x1004739    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie+0x59            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:157

#    0x10064a3    main.main+0x6a3                                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:115

#    0x42d1fc    runtime.main+0x20c                                                    /usr/local/go/1.9/go/src/runtime/proc.go:185

1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xff0725 0xff552a 0x8c2ecf 0x45c191

#    0xff0724    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run+0x94                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545

#    0xff5529    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:387

#    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71

1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0x8aac9c 0x8b6141 0x6e6234 0x6e5f25 0x6e58c8 0xa1c178 0xe78795 0xe78e4c 0x8c4658 0x45c191

#    0x4402bf    sync.runtime_notifyListWait+0x10f                                                /usr/local/go/1.9/go/src/runtime/sema.go:507

#    0x46ffaf    sync.(*Cond).Wait+0x7f                                                        /usr/local/go/1.9/go/src/sync/cond.go:56

#    0x8aac9b    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*pipe).Read+0x9b                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/pipe.go:64

#    0x8b6140    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read+0xb0                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1674

#    0x6e6233    encoding/json.(*Decoder).refill+0x113                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:152

#    0x6e5f24    encoding/json.(*Decoder).readValue+0x274                                            /usr/local/go/1.9/go/src/encoding/json/stream.go:128

#    0x6e58c7    encoding/json.(*Decoder).Decode+0x77                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:57

#    0xa1c177    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x2c7        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150

#    0xe78794    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0x94    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77

#    0xe78e4b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x7b                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49

#    0x8c4657    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93

1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xfe7736 0xfe51c0 0xff533a 0x8c2f5e 0x8c27cd 0x8c26fd 0xfe4fe7 0xfedf9a 0x45c191

#    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507

#    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56

#    0xfe7735    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0xb5                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438

#    0xfe51bf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:150

#    0xff5339    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124

#    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133

#    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134

#    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

#    0xfe4fe6    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x2b6                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124

#    0xfedf99    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x4c9                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:226

1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xffac75 0xffb6bc 0xffbbec 0xffdddd 0xffdd7b 0xffe92a 0x8c2f5e 0x8c27cd 0x8c26fd 0x45c191

#    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507

#    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56

#    0xffac74    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*Type).Get+0x94                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:124

#    0xffb6bb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get+0x3b                                            <autogenerated>:1

#    0xffbbeb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get+0x3b                                        <autogenerated>:1

#    0xffdddc    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).processNextItem+0x3c                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:43

#    0xffdd7a    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).runWorker+0x2a                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:37

#    0xffe929    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).(github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.runWorker)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:88

#    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133

#    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134

#    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

1 @ 0x42d69c 0x43d9f8 0x692f95 0x45c191

#    0x692f94    net/http.(*persistConn).writeLoop+0x164    /usr/local/go/1.9/go/src/net/http/transport.go:1759

1 @ 0x42d69c 0x43d9f8 0xfc61b6 0xfda64b 0x45c191

#    0xfc61b5    github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run+0x235    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:294

#    0xfda64a    github.com/couchbase/couchbase-operator/pkg/cluster.New.func1+0x27a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150

1 @ 0x42d69c 0x43d9f8 0xfebcd8 0xfeb64c 0xff45b3 0x8c2f5e 0x8c27cd 0x8c26fd 0xfea514 0xff52f4 0x8c2e11 0x8c2ecf 0x45c191

#    0xfebcd7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x287        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:373

#    0xfeb64b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0xffb        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:339

#    0xff45b2    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x32            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:204

#    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133

#    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134

#    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

#    0xfea513    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run+0x163            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:203

#    0xff52f3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm+0x33            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:122

#    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54

#    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71

1 @ 0x42d69c 0x43d9f8 0xff04d0 0xff556a 0x8c2ecf 0x45c191

#    0xff04cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop+0x18f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:519

#    0xff5569    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.pop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388

#    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71

1 @ 0x42d69c 0x43d9f8 0xff47d0 0x45c191

#    0xff47cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1+0x18f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:278

1 @ 0x42d69c 0x43d9f8 0xff95ef 0x45c191

#    0xff95ee    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x39e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:208

1 @ 0x7071b2 0x706fb0 0x703b0b 0x70fc48 0x70ff4b 0x67e9d4 0x680080 0x6815f4 0x67d8ad 0x45c191

#    0x7071b1    runtime/pprof.writeRuntimeProfile+0xa1    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:637

#    0x706faf    runtime/pprof.writeGoroutine+0x9f    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:599

#    0x703b0a    runtime/pprof.(*Profile).WriteTo+0x3aa    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:310

#    0x70fc47    net/http/pprof.handler.ServeHTTP+0x1b7    /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:237

#    0x70ff4a    net/http/pprof.Index+0x1da        /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:248

#    0x67e9d3    net/http.HandlerFunc.ServeHTTP+0x43    /usr/local/go/1.9/go/src/net/http/server.go:1918

#    0x68007f    net/http.(*ServeMux).ServeHTTP+0x12f    /usr/local/go/1.9/go/src/net/http/server.go:2254

#    0x6815f3    net/http.serverHandler.ServeHTTP+0xb3    /usr/local/go/1.9/go/src/net/http/server.go:2619

#    0x67d8ac    net/http.(*conn).serve+0x71c        /usr/local/go/1.9/go/src/net/http/server.go:1801

Ashwin Govindarajulu added a comment - 10/Aug/18 3:00 AM /tmp $ cat cluster {"clusters":{"test-couchbase-gcjn8":{"last_reconcile_loop_start_time":"2018-08-10T09:56:50.534193208Z","last_reconcile_loop_end_time":"2018-08-10T09:56:50.579002946Z","last_completed_loop_duration_sec":0.044809768,"reconcile_loop_status":"sleeping","reconcile_loop_sleep_time":8,"clusterPhase":"Running","controlPaused":false}}} /tmp $ cat goroutine?debug\=1 goroutine profile: total 200 105 @ 0x42d69c 0x42d78e 0x403d2b 0x403ab3 0x8c463e 0x45c191 # 0x8c463d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:114 37 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xfc2ebc 0x45c191 # 0xfc2ebb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0xab /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record/event.go:229 36 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c3d51 0x45c191 # 0x8c3d50 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x60 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208 2 @ 0x42d69c 0x43d9f8 0x8c3127 0x45c191 # 0x8c3126 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1+0x196 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354 1 @ 0x42d69c 0x40502b 0x4047fb 0xff4554 0x45c191 # 0xff4553 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:103 1 @ 0x42d69c 0x40502b 0x4047fb 0xffd8af 0x45c191 # 0xffd8ae github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).Run+0x2ee /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:90 1 @ 0x42d69c 0x40502b 0x4047fb 0xfff000 0xfffd34 0x100675f 0x45c191 # 0xffefff github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk.Run+0xbf /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/api.go:69 # 0xfffd33 github.com/couchbase/couchbase-operator/pkg/controller.(*Controller).Start+0x303 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:95 # 0x100675e main.run+0x12e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:136 1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x5d0b45 0x5d10b0 0x5d4c00 0x53265b 0x4736c6 0x473838 0x8a213b 0x8a2aa4 0x8b40d0 0x8b38fd 0x45c191 # 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173 # 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 # 0x497259 internal/poll.(*FD).Read+0x189 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125 # 0x57d351 net.(*netFD).Read+0x51 /usr/local/go/1.9/go/src/net/fd_unix.go:202 # 0x58f8ac net.(*conn).Read+0x6c /usr/local/go/1.9/go/src/net/net.go:176 # 0x5d0b44 crypto/tls.(*block).readFromUntil+0x94 /usr/local/go/1.9/go/src/crypto/tls/conn.go:488 # 0x5d10af crypto/tls.(*Conn).readRecord+0xdf /usr/local/go/1.9/go/src/crypto/tls/conn.go:590 # 0x5d4bff crypto/tls.(*Conn).Read+0x10f /usr/local/go/1.9/go/src/crypto/tls/conn.go:1134 # 0x53265a bufio.(*Reader).Read+0x30a /usr/local/go/1.9/go/src/bufio/bufio.go:213 # 0x4736c5 io.ReadAtLeast+0x85 /usr/local/go/1.9/go/src/io/io.go:309 # 0x473837 io.ReadFull+0x57 /usr/local/go/1.9/go/src/io/io.go:327 # 0x8a213a github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.readFrameHeader+0x7a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:237 # 0x8a2aa3 github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:492 # 0x8b40cf github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run+0x8f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1428 # 0x8b38fc github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop+0x9c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1354 1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x690d10 0x531e8a 0x531fda 0x6917e5 0x45c191 # 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173 # 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 # 0x497259 internal/poll.(*FD).Read+0x189 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125 # 0x57d351 net.(*netFD).Read+0x51 /usr/local/go/1.9/go/src/net/fd_unix.go:202 # 0x58f8ac net.(*conn).Read+0x6c /usr/local/go/1.9/go/src/net/net.go:176 # 0x690d0f net/http.(*persistConn).Read+0x13f /usr/local/go/1.9/go/src/net/http/transport.go:1391 # 0x531e89 bufio.(*Reader).fill+0x119 /usr/local/go/1.9/go/src/bufio/bufio.go:97 # 0x531fd9 bufio.(*Reader).Peek+0x39 /usr/local/go/1.9/go/src/bufio/bufio.go:129 # 0x6917e4 net/http.(*persistConn).readLoop+0x184 /usr/local/go/1.9/go/src/net/http/transport.go:1539 1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x498912 0x57dd72 0x599aee 0x597ec9 0x682cdf 0x681982 0x6816c9 0x68285f 0x45c191 # 0x427dd6 internal/poll.runtime_pollWait+0x56 /usr/local/go/1.9/go/src/runtime/netpoll.go:173 # 0x49643d internal/poll.(*pollDesc).wait+0xad /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4964bc internal/poll.(*pollDesc).waitRead+0x3c /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 # 0x498911 internal/poll.(*FD).Accept+0x1e1 /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:334 # 0x57dd71 net.(*netFD).accept+0x41 /usr/local/go/1.9/go/src/net/fd_unix.go:238 # 0x599aed net.(*TCPListener).accept+0x2d /usr/local/go/1.9/go/src/net/tcpsock_posix.go:136 # 0x597ec8 net.(*TCPListener).AcceptTCP+0x48 /usr/local/go/1.9/go/src/net/tcpsock.go:234 # 0x682cde net/http.tcpKeepAliveListener.Accept+0x2e /usr/local/go/1.9/go/src/net/http/server.go:3120 # 0x681981 net/http.(*Server).Serve+0x1b1 /usr/local/go/1.9/go/src/net/http/server.go:2695 # 0x6816c8 net/http.(*Server).ListenAndServe+0xa8 /usr/local/go/1.9/go/src/net/http/server.go:2636 # 0x68285e net/http.ListenAndServe+0x7e /usr/local/go/1.9/go/src/net/http/server.go:2882 1 @ 0x42d69c 0x42d78e 0x404b54 0x4047fb 0xfefbf6 0xff54e4 0x8c2e11 0x8c2ecf 0x45c191 # 0xfefbf5 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x45 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:427 # 0xff54e3 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:219 # 0x8c2e10 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 # 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x860c2f 0x45c191 # 0x860c2e github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon+0x9e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog/glog.go:879 1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c2c18 0x8c2af4 0x8c2a2d 0x10057d8 0x8c2f5e 0x8c27cd 0x8c26fd 0x1004972 0x10046a0 0x100473a 0x10064a4 0x42d1fd 0x45c191 # 0x8c2c17 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor+0x57 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:310 # 0x8c2af3 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal+0x93 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:227 # 0x8c2a2c github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Poll+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:221 # 0x10057d7 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1+0x97 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:193 # 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 # 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 # 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 # 0x1004971 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x71 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:192 # 0x100469f github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x9f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:146 # 0x1004739 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie+0x59 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:157 # 0x10064a3 main.main+0x6a3 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:115 # 0x42d1fc runtime.main+0x20c /usr/local/go/1.9/go/src/runtime/proc.go:185 1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xff0725 0xff552a 0x8c2ecf 0x45c191 # 0xff0724 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 # 0xff5529 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:387 # 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0x8aac9c 0x8b6141 0x6e6234 0x6e5f25 0x6e58c8 0xa1c178 0xe78795 0xe78e4c 0x8c4658 0x45c191 # 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507 # 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56 # 0x8aac9b github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*pipe).Read+0x9b /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/pipe.go:64 # 0x8b6140 github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read+0xb0 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1674 # 0x6e6233 encoding/json.(*Decoder).refill+0x113 /usr/local/go/1.9/go/src/encoding/json/stream.go:152 # 0x6e5f24 encoding/json.(*Decoder).readValue+0x274 /usr/local/go/1.9/go/src/encoding/json/stream.go:128 # 0x6e58c7 encoding/json.(*Decoder).Decode+0x77 /usr/local/go/1.9/go/src/encoding/json/stream.go:57 # 0xa1c177 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x2c7 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 # 0xe78794 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 # 0xe78e4b github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x7b /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49 # 0x8c4657 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93 1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xfe7736 0xfe51c0 0xff533a 0x8c2f5e 0x8c27cd 0x8c26fd 0xfe4fe7 0xfedf9a 0x45c191 # 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507 # 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56 # 0xfe7735 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0xb5 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438 # 0xfe51bf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:150 # 0xff5339 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124 # 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 # 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 # 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 # 0xfe4fe6 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x2b6 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124 # 0xfedf99 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x4c9 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:226 1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xffac75 0xffb6bc 0xffbbec 0xffdddd 0xffdd7b 0xffe92a 0x8c2f5e 0x8c27cd 0x8c26fd 0x45c191 # 0x4402bf sync.runtime_notifyListWait+0x10f /usr/local/go/1.9/go/src/runtime/sema.go:507 # 0x46ffaf sync.(*Cond).Wait+0x7f /usr/local/go/1.9/go/src/sync/cond.go:56 # 0xffac74 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*Type).Get+0x94 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:124 # 0xffb6bb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get+0x3b <autogenerated>:1 # 0xffbbeb github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get+0x3b <autogenerated>:1 # 0xffdddc github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).processNextItem+0x3c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:43 # 0xffdd7a github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).runWorker+0x2a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:37 # 0xffe929 github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).(github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.runWorker)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:88 # 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 # 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 # 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 1 @ 0x42d69c 0x43d9f8 0x692f95 0x45c191 # 0x692f94 net/http.(*persistConn).writeLoop+0x164 /usr/local/go/1.9/go/src/net/http/transport.go:1759 1 @ 0x42d69c 0x43d9f8 0xfc61b6 0xfda64b 0x45c191 # 0xfc61b5 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run+0x235 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:294 # 0xfda64a github.com/couchbase/couchbase-operator/pkg/cluster.New.func1+0x27a /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150 1 @ 0x42d69c 0x43d9f8 0xfebcd8 0xfeb64c 0xff45b3 0x8c2f5e 0x8c27cd 0x8c26fd 0xfea514 0xff52f4 0x8c2e11 0x8c2ecf 0x45c191 # 0xfebcd7 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x287 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:373 # 0xfeb64b github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0xffb /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:339 # 0xff45b2 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x32 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:204 # 0x8c2f5d github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 # 0x8c27cc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 # 0x8c26fc github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 # 0xfea513 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run+0x163 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:203 # 0xff52f3 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm+0x33 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:122 # 0x8c2e10 github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 # 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 1 @ 0x42d69c 0x43d9f8 0xff04d0 0xff556a 0x8c2ecf 0x45c191 # 0xff04cf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop+0x18f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:519 # 0xff5569 github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.pop)-fm+0x29 /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388 # 0x8c2ece github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 1 @ 0x42d69c 0x43d9f8 0xff47d0 0x45c191 # 0xff47cf github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1+0x18f /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:278 1 @ 0x42d69c 0x43d9f8 0xff95ef 0x45c191 # 0xff95ee github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x39e /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:208 1 @ 0x7071b2 0x706fb0 0x703b0b 0x70fc48 0x70ff4b 0x67e9d4 0x680080 0x6815f4 0x67d8ad 0x45c191 # 0x7071b1 runtime/pprof.writeRuntimeProfile+0xa1 /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:637 # 0x706faf runtime/pprof.writeGoroutine+0x9f /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:599 # 0x703b0a runtime/pprof.(*Profile).WriteTo+0x3aa /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:310 # 0x70fc47 net/http/pprof.handler.ServeHTTP+0x1b7 /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:237 # 0x70ff4a net/http/pprof.Index+0x1da /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:248 # 0x67e9d3 net/http.HandlerFunc.ServeHTTP+0x43 /usr/local/go/1.9/go/src/net/http/server.go:1918 # 0x68007f net/http.(*ServeMux).ServeHTTP+0x12f /usr/local/go/1.9/go/src/net/http/server.go:2254 # 0x6815f3 net/http.serverHandler.ServeHTTP+0xb3 /usr/local/go/1.9/go/src/net/http/server.go:2619 # 0x67d8ac net/http.(*conn).serve+0x71c /usr/local/go/1.9/go/src/net/http/server.go:1801

Simon Murray added a comment - 10/Aug/18 3:22 AM

Okay so it's 11:21+0100 now, which means in UTC that's 10:21. You added that comment 21 minutes ago, so 10:00 and the time stamp says the last reconcile was 09:56. Do you agree it is working and not hanging?

Simon Murray added a comment - 10/Aug/18 3:22 AM Okay so it's 11:21+0100 now, which means in UTC that's 10:21. You added that comment 21 minutes ago, so 10:00 and the time stamp says the last reconcile was 09:56. Do you agree it is working and not hanging?

Ashwin Govindarajulu added a comment - 10/Aug/18 3:35 AM

Yeah. I too agree that there is no problem with the operator now.

But there is something wrong in the AWS node on which the testing is running.

I have restarted the test now. Will keep you posted on further updates.

Ashwin Govindarajulu added a comment - 10/Aug/18 3:35 AM Yeah. I too agree that there is no problem with the operator now. But there is something wrong in the AWS node on which the testing is running. I have restarted the test now. Will keep you posted on further updates.

Ashwin Govindarajulu added a comment - 13/Aug/18 10:53 PM

Not seeing this problem now. Looks like there was some runtime related issues.

After rebooting the target node, tests seems to work fine.

Ashwin Govindarajulu added a comment - 13/Aug/18 10:53 PM Not seeing this problem now. Looks like there was some runtime related issues. After rebooting the target node, tests seems to work fine.

People

Assignee:: Ashwin Govindarajulu

Reporter:: Ashwin Govindarajulu

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 09/Aug/18 11:27 PM

Updated:: 13/Aug/18 10:53 PM

Resolved:: 13/Aug/18 10:53 PM

Gerrit Reviews

There are no open Gerrit changes

PagerDuty

Couchbase Kubernetes

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty