Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-524

Operator get stalled when performing resize operations in couchbase cluster

    XMLWordPrintable

Details

    Description

      Operator gets stalled while performing resize related testing on openshift platform running on AWS.

      As per the operator logs, operator logs started from 2018-08-10T03:23:01Z to 2018-08-10T05:32:02Z

      But operator pod is running more than 3hrs as per kube data.

      [ec2-user@ip-172-16-0-187 ~]$ kubectl get pods
      NAME                                  READY     STATUS    RESTARTS   AGE
      couchbase-operator-554b978699-tl49x   1/1       Running   0          3h
      docker-registry-1-9fznr               1/1       Running   1          2d
      registry-console-1-jcn85              1/1       Running   1          2d
      router-1-mkhbw                        1/1       Running   1          2d
      test-couchbase-gcjn8-0000             1/1       Running   0          54m

       

      [ec2-user@ip-172-16-0-187 ~]$ kubectl get nodes
      NAME                                         STATUS    ROLES     AGE       VERSION
      ip-172-16-0-134.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
      ip-172-16-0-187.us-west-2.compute.internal   Ready     master    7d        v1.9.1+a0ce1bc657
      ip-172-16-0-229.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
      ip-172-16-0-239.us-west-2.compute.internal   Ready     compute   7d        v1.9.1+a0ce1bc657
      [ec2-user@ip-172-16-0-187 ~]${noformat}

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment -

          Not enough info to debug.  Can you exec into the operator pod and run:

          wget localhost:8080/debug/pprof/goroutine?debug=1

          then post the output.

          simon.murray Simon Murray added a comment - Not enough info to debug.  Can you exec into the operator pod and run: wget localhost:8080/debug/pprof/goroutine?debug=1 then post the output.
          simon.murray Simon Murray added a comment -

          Opened K8S-527 to perform this step automatically in the 1.1.0 release

          simon.murray Simon Murray added a comment - Opened K8S-527 to perform this step automatically in the 1.1.0 release

          I am getting Permission denied error code

          ec2-user@ip-172-16-0-187 ~]$ kubectl exec couchbase-operator-554b978699-tl49x wget localhost:8080/debug/pprof/goroutine?debug=1
          Connecting to localhost:8080 (127.0.0.1:8080)
          wget: can't open 'goroutine?debug=1': Permission denied
          command terminated with exit code 1

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - I am getting Permission denied error code ec2-user@ip-172-16-0-187 ~]$ kubectl exec couchbase-operator-554b978699-tl49x wget localhost:8080/debug/pprof/goroutine?debug=1 Connecting to localhost:8080 (127.0.0.1:8080) wget: can't open 'goroutine?debug=1': Permission denied command terminated with exit code 1
          simon.murray Simon Murray added a comment -

          That's a file system write error.   You may also wish to look at the /v1/stats/cluster end point too as that contains runloop stats

          simon.murray Simon Murray added a comment - That's a file system write error.   You may also wish to look at the /v1/stats/cluster end point too as that contains runloop stats

          /tmp $ cat cluster
          {"clusters":{"test-couchbase-gcjn8":{"last_reconcile_loop_start_time":"2018-08-10T09:56:50.534193208Z","last_reconcile_loop_end_time":"2018-08-10T09:56:50.579002946Z","last_completed_loop_duration_sec":0.044809768,"reconcile_loop_status":"sleeping","reconcile_loop_sleep_time":8,"clusterPhase":"Running","controlPaused":false}}}
           
          /tmp $ cat goroutine?debug\=1
          goroutine profile: total 200
          105 @ 0x42d69c 0x42d78e 0x403d2b 0x403ab3 0x8c463e 0x45c191
          #    0x8c463d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11d    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:114
           
          37 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xfc2ebc 0x45c191
          #    0xfc2ebb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0xab    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record/event.go:229
           
          36 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c3d51 0x45c191
          #    0x8c3d50    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x60    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208
           
          2 @ 0x42d69c 0x43d9f8 0x8c3127 0x45c191
          #    0x8c3126    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1+0x196    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354
           
          1 @ 0x42d69c 0x40502b 0x4047fb 0xff4554 0x45c191
          #    0xff4553    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:103
           
          1 @ 0x42d69c 0x40502b 0x4047fb 0xffd8af 0x45c191
          #    0xffd8ae    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).Run+0x2ee    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:90
           
          1 @ 0x42d69c 0x40502b 0x4047fb 0xfff000 0xfffd34 0x100675f 0x45c191
          #    0xffefff    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk.Run+0xbf    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/api.go:69
          #    0xfffd33    github.com/couchbase/couchbase-operator/pkg/controller.(*Controller).Start+0x303        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:95
          #    0x100675e    main.run+0x12e                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:136
           
          1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x5d0b45 0x5d10b0 0x5d4c00 0x53265b 0x4736c6 0x473838 0x8a213b 0x8a2aa4 0x8b40d0 0x8b38fd 0x45c191
          #    0x427dd6    internal/poll.runtime_pollWait+0x56                                    /usr/local/go/1.9/go/src/runtime/netpoll.go:173
          #    0x49643d    internal/poll.(*pollDesc).wait+0xad                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
          #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
          #    0x497259    internal/poll.(*FD).Read+0x189                                        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125
          #    0x57d351    net.(*netFD).Read+0x51                                            /usr/local/go/1.9/go/src/net/fd_unix.go:202
          #    0x58f8ac    net.(*conn).Read+0x6c                                            /usr/local/go/1.9/go/src/net/net.go:176
          #    0x5d0b44    crypto/tls.(*block).readFromUntil+0x94                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:488
          #    0x5d10af    crypto/tls.(*Conn).readRecord+0xdf                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:590
          #    0x5d4bff    crypto/tls.(*Conn).Read+0x10f                                        /usr/local/go/1.9/go/src/crypto/tls/conn.go:1134
          #    0x53265a    bufio.(*Reader).Read+0x30a                                        /usr/local/go/1.9/go/src/bufio/bufio.go:213
          #    0x4736c5    io.ReadAtLeast+0x85                                            /usr/local/go/1.9/go/src/io/io.go:309
          #    0x473837    io.ReadFull+0x57                                            /usr/local/go/1.9/go/src/io/io.go:327
          #    0x8a213a    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.readFrameHeader+0x7a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:237
          #    0x8a2aa3    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:492
          #    0x8b40cf    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run+0x8f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1428
          #    0x8b38fc    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop+0x9c    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1354
           
          1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x690d10 0x531e8a 0x531fda 0x6917e5 0x45c191
          #    0x427dd6    internal/poll.runtime_pollWait+0x56    /usr/local/go/1.9/go/src/runtime/netpoll.go:173
          #    0x49643d    internal/poll.(*pollDesc).wait+0xad    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
          #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
          #    0x497259    internal/poll.(*FD).Read+0x189        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125
          #    0x57d351    net.(*netFD).Read+0x51            /usr/local/go/1.9/go/src/net/fd_unix.go:202
          #    0x58f8ac    net.(*conn).Read+0x6c            /usr/local/go/1.9/go/src/net/net.go:176
          #    0x690d0f    net/http.(*persistConn).Read+0x13f    /usr/local/go/1.9/go/src/net/http/transport.go:1391
          #    0x531e89    bufio.(*Reader).fill+0x119        /usr/local/go/1.9/go/src/bufio/bufio.go:97
          #    0x531fd9    bufio.(*Reader).Peek+0x39        /usr/local/go/1.9/go/src/bufio/bufio.go:129
          #    0x6917e4    net/http.(*persistConn).readLoop+0x184    /usr/local/go/1.9/go/src/net/http/transport.go:1539
           
          1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x498912 0x57dd72 0x599aee 0x597ec9 0x682cdf 0x681982 0x6816c9 0x68285f 0x45c191
          #    0x427dd6    internal/poll.runtime_pollWait+0x56        /usr/local/go/1.9/go/src/runtime/netpoll.go:173
          #    0x49643d    internal/poll.(*pollDesc).wait+0xad        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85
          #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90
          #    0x498911    internal/poll.(*FD).Accept+0x1e1        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:334
          #    0x57dd71    net.(*netFD).accept+0x41            /usr/local/go/1.9/go/src/net/fd_unix.go:238
          #    0x599aed    net.(*TCPListener).accept+0x2d            /usr/local/go/1.9/go/src/net/tcpsock_posix.go:136
          #    0x597ec8    net.(*TCPListener).AcceptTCP+0x48        /usr/local/go/1.9/go/src/net/tcpsock.go:234
          #    0x682cde    net/http.tcpKeepAliveListener.Accept+0x2e    /usr/local/go/1.9/go/src/net/http/server.go:3120
          #    0x681981    net/http.(*Server).Serve+0x1b1            /usr/local/go/1.9/go/src/net/http/server.go:2695
          #    0x6816c8    net/http.(*Server).ListenAndServe+0xa8        /usr/local/go/1.9/go/src/net/http/server.go:2636
          #    0x68285e    net/http.ListenAndServe+0x7e            /usr/local/go/1.9/go/src/net/http/server.go:2882
           
          1 @ 0x42d69c 0x42d78e 0x404b54 0x4047fb 0xfefbf6 0xff54e4 0x8c2e11 0x8c2ecf 0x45c191
          #    0xfefbf5    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x45                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:427
          #    0xff54e3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:219
          #    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30                                    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
          #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
           
          1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x860c2f 0x45c191
          #    0x860c2e    github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon+0x9e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog/glog.go:879
           
          1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c2c18 0x8c2af4 0x8c2a2d 0x10057d8 0x8c2f5e 0x8c27cd 0x8c26fd 0x1004972 0x10046a0 0x100473a 0x10064a4 0x42d1fd 0x45c191
          #    0x8c2c17    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor+0x57                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:310
          #    0x8c2af3    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal+0x93            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:227
          #    0x8c2a2c    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Poll+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:221
          #    0x10057d7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1+0x97    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:193
          #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
          #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
          #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
          #    0x1004971    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x71    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:192
          #    0x100469f    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x9f        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:146
          #    0x1004739    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie+0x59            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:157
          #    0x10064a3    main.main+0x6a3                                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:115
          #    0x42d1fc    runtime.main+0x20c                                                    /usr/local/go/1.9/go/src/runtime/proc.go:185
           
          1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xff0725 0xff552a 0x8c2ecf 0x45c191
          #    0xff0724    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run+0x94                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545
          #    0xff5529    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:387
          #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
           
          1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0x8aac9c 0x8b6141 0x6e6234 0x6e5f25 0x6e58c8 0xa1c178 0xe78795 0xe78e4c 0x8c4658 0x45c191
          #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                /usr/local/go/1.9/go/src/runtime/sema.go:507
          #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                        /usr/local/go/1.9/go/src/sync/cond.go:56
          #    0x8aac9b    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*pipe).Read+0x9b                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/pipe.go:64
          #    0x8b6140    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read+0xb0                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1674
          #    0x6e6233    encoding/json.(*Decoder).refill+0x113                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:152
          #    0x6e5f24    encoding/json.(*Decoder).readValue+0x274                                            /usr/local/go/1.9/go/src/encoding/json/stream.go:128
          #    0x6e58c7    encoding/json.(*Decoder).Decode+0x77                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:57
          #    0xa1c177    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x2c7        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150
          #    0xe78794    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0x94    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77
          #    0xe78e4b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x7b                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49
          #    0x8c4657    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93
           
          1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xfe7736 0xfe51c0 0xff533a 0x8c2f5e 0x8c27cd 0x8c26fd 0xfe4fe7 0xfedf9a 0x45c191
          #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507
          #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56
          #    0xfe7735    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0xb5                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438
          #    0xfe51bf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:150
          #    0xff5339    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124
          #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
          #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
          #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
          #    0xfe4fe6    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x2b6                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124
          #    0xfedf99    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x4c9                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:226
           
          1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xffac75 0xffb6bc 0xffbbec 0xffdddd 0xffdd7b 0xffe92a 0x8c2f5e 0x8c27cd 0x8c26fd 0x45c191
          #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507
          #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56
          #    0xffac74    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*Type).Get+0x94                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:124
          #    0xffb6bb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get+0x3b                                            <autogenerated>:1
          #    0xffbbeb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get+0x3b                                        <autogenerated>:1
          #    0xffdddc    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).processNextItem+0x3c                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:43
          #    0xffdd7a    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).runWorker+0x2a                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:37
          #    0xffe929    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).(github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.runWorker)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:88
          #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
          #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
          #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
           
          1 @ 0x42d69c 0x43d9f8 0x692f95 0x45c191
          #    0x692f94    net/http.(*persistConn).writeLoop+0x164    /usr/local/go/1.9/go/src/net/http/transport.go:1759
           
          1 @ 0x42d69c 0x43d9f8 0xfc61b6 0xfda64b 0x45c191
          #    0xfc61b5    github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run+0x235    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:294
          #    0xfda64a    github.com/couchbase/couchbase-operator/pkg/cluster.New.func1+0x27a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150
           
          1 @ 0x42d69c 0x43d9f8 0xfebcd8 0xfeb64c 0xff45b3 0x8c2f5e 0x8c27cd 0x8c26fd 0xfea514 0xff52f4 0x8c2e11 0x8c2ecf 0x45c191
          #    0xfebcd7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x287        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:373
          #    0xfeb64b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0xffb        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:339
          #    0xff45b2    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x32            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:204
          #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
          #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
          #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
          #    0xfea513    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run+0x163            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:203
          #    0xff52f3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm+0x33            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:122
          #    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
          #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
           
          1 @ 0x42d69c 0x43d9f8 0xff04d0 0xff556a 0x8c2ecf 0x45c191
          #    0xff04cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop+0x18f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:519
          #    0xff5569    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.pop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388
          #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
           
          1 @ 0x42d69c 0x43d9f8 0xff47d0 0x45c191
          #    0xff47cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1+0x18f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:278
           
          1 @ 0x42d69c 0x43d9f8 0xff95ef 0x45c191
          #    0xff95ee    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x39e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:208
           
          1 @ 0x7071b2 0x706fb0 0x703b0b 0x70fc48 0x70ff4b 0x67e9d4 0x680080 0x6815f4 0x67d8ad 0x45c191
          #    0x7071b1    runtime/pprof.writeRuntimeProfile+0xa1    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:637
          #    0x706faf    runtime/pprof.writeGoroutine+0x9f    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:599
          #    0x703b0a    runtime/pprof.(*Profile).WriteTo+0x3aa    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:310
          #    0x70fc47    net/http/pprof.handler.ServeHTTP+0x1b7    /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:237
          #    0x70ff4a    net/http/pprof.Index+0x1da        /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:248
          #    0x67e9d3    net/http.HandlerFunc.ServeHTTP+0x43    /usr/local/go/1.9/go/src/net/http/server.go:1918
          #    0x68007f    net/http.(*ServeMux).ServeHTTP+0x12f    /usr/local/go/1.9/go/src/net/http/server.go:2254
          #    0x6815f3    net/http.serverHandler.ServeHTTP+0xb3    /usr/local/go/1.9/go/src/net/http/server.go:2619
          #    0x67d8ac    net/http.(*conn).serve+0x71c        /usr/local/go/1.9/go/src/net/http/server.go:1801
          

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - /tmp $ cat cluster {"clusters":{"test-couchbase-gcjn8":{"last_reconcile_loop_start_time":"2018-08-10T09:56:50.534193208Z","last_reconcile_loop_end_time":"2018-08-10T09:56:50.579002946Z","last_completed_loop_duration_sec":0.044809768,"reconcile_loop_status":"sleeping","reconcile_loop_sleep_time":8,"clusterPhase":"Running","controlPaused":false}}}   /tmp $ cat goroutine?debug\=1 goroutine profile: total 200 105 @ 0x42d69c 0x42d78e 0x403d2b 0x403ab3 0x8c463e 0x45c191 #    0x8c463d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x11d    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:114   37 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xfc2ebc 0x45c191 #    0xfc2ebb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1+0xab    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/record/event.go:229   36 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c3d51 0x45c191 #    0x8c3d50    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop+0x60    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:208   2 @ 0x42d69c 0x43d9f8 0x8c3127 0x45c191 #    0x8c3126    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1+0x196    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354   1 @ 0x42d69c 0x40502b 0x4047fb 0xff4554 0x45c191 #    0xff4553    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:103   1 @ 0x42d69c 0x40502b 0x4047fb 0xffd8af 0x45c191 #    0xffd8ae    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).Run+0x2ee    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:90   1 @ 0x42d69c 0x40502b 0x4047fb 0xfff000 0xfffd34 0x100675f 0x45c191 #    0xffefff    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk.Run+0xbf    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/api.go:69 #    0xfffd33    github.com/couchbase/couchbase-operator/pkg/controller.(*Controller).Start+0x303        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:95 #    0x100675e    main.run+0x12e                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:136   1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x5d0b45 0x5d10b0 0x5d4c00 0x53265b 0x4736c6 0x473838 0x8a213b 0x8a2aa4 0x8b40d0 0x8b38fd 0x45c191 #    0x427dd6    internal/poll.runtime_pollWait+0x56                                    /usr/local/go/1.9/go/src/runtime/netpoll.go:173 #    0x49643d    internal/poll.(*pollDesc).wait+0xad                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c                                    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 #    0x497259    internal/poll.(*FD).Read+0x189                                        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125 #    0x57d351    net.(*netFD).Read+0x51                                            /usr/local/go/1.9/go/src/net/fd_unix.go:202 #    0x58f8ac    net.(*conn).Read+0x6c                                            /usr/local/go/1.9/go/src/net/net.go:176 #    0x5d0b44    crypto/tls.(*block).readFromUntil+0x94                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:488 #    0x5d10af    crypto/tls.(*Conn).readRecord+0xdf                                    /usr/local/go/1.9/go/src/crypto/tls/conn.go:590 #    0x5d4bff    crypto/tls.(*Conn).Read+0x10f                                        /usr/local/go/1.9/go/src/crypto/tls/conn.go:1134 #    0x53265a    bufio.(*Reader).Read+0x30a                                        /usr/local/go/1.9/go/src/bufio/bufio.go:213 #    0x4736c5    io.ReadAtLeast+0x85                                            /usr/local/go/1.9/go/src/io/io.go:309 #    0x473837    io.ReadFull+0x57                                            /usr/local/go/1.9/go/src/io/io.go:327 #    0x8a213a    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.readFrameHeader+0x7a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:237 #    0x8a2aa3    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/frame.go:492 #    0x8b40cf    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run+0x8f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1428 #    0x8b38fc    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*ClientConn).readLoop+0x9c    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1354   1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x49725a 0x57d352 0x58f8ad 0x690d10 0x531e8a 0x531fda 0x6917e5 0x45c191 #    0x427dd6    internal/poll.runtime_pollWait+0x56    /usr/local/go/1.9/go/src/runtime/netpoll.go:173 #    0x49643d    internal/poll.(*pollDesc).wait+0xad    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c    /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 #    0x497259    internal/poll.(*FD).Read+0x189        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:125 #    0x57d351    net.(*netFD).Read+0x51            /usr/local/go/1.9/go/src/net/fd_unix.go:202 #    0x58f8ac    net.(*conn).Read+0x6c            /usr/local/go/1.9/go/src/net/net.go:176 #    0x690d0f    net/http.(*persistConn).Read+0x13f    /usr/local/go/1.9/go/src/net/http/transport.go:1391 #    0x531e89    bufio.(*Reader).fill+0x119        /usr/local/go/1.9/go/src/bufio/bufio.go:97 #    0x531fd9    bufio.(*Reader).Peek+0x39        /usr/local/go/1.9/go/src/bufio/bufio.go:129 #    0x6917e4    net/http.(*persistConn).readLoop+0x184    /usr/local/go/1.9/go/src/net/http/transport.go:1539   1 @ 0x42d69c 0x4287da 0x427dd7 0x49643e 0x4964bd 0x498912 0x57dd72 0x599aee 0x597ec9 0x682cdf 0x681982 0x6816c9 0x68285f 0x45c191 #    0x427dd6    internal/poll.runtime_pollWait+0x56        /usr/local/go/1.9/go/src/runtime/netpoll.go:173 #    0x49643d    internal/poll.(*pollDesc).wait+0xad        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:85 #    0x4964bc    internal/poll.(*pollDesc).waitRead+0x3c        /usr/local/go/1.9/go/src/internal/poll/fd_poll_runtime.go:90 #    0x498911    internal/poll.(*FD).Accept+0x1e1        /usr/local/go/1.9/go/src/internal/poll/fd_unix.go:334 #    0x57dd71    net.(*netFD).accept+0x41            /usr/local/go/1.9/go/src/net/fd_unix.go:238 #    0x599aed    net.(*TCPListener).accept+0x2d            /usr/local/go/1.9/go/src/net/tcpsock_posix.go:136 #    0x597ec8    net.(*TCPListener).AcceptTCP+0x48        /usr/local/go/1.9/go/src/net/tcpsock.go:234 #    0x682cde    net/http.tcpKeepAliveListener.Accept+0x2e    /usr/local/go/1.9/go/src/net/http/server.go:3120 #    0x681981    net/http.(*Server).Serve+0x1b1            /usr/local/go/1.9/go/src/net/http/server.go:2695 #    0x6816c8    net/http.(*Server).ListenAndServe+0xa8        /usr/local/go/1.9/go/src/net/http/server.go:2636 #    0x68285e    net/http.ListenAndServe+0x7e            /usr/local/go/1.9/go/src/net/http/server.go:2882   1 @ 0x42d69c 0x42d78e 0x404b54 0x4047fb 0xfefbf6 0xff54e4 0x8c2e11 0x8c2ecf 0x45c191 #    0xfefbf5    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).run+0x45                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:427 #    0xff54e3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedProcessor).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x33    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:219 #    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30                                    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71   1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x860c2f 0x45c191 #    0x860c2e    github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog.(*loggingT).flushDaemon+0x9e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/golang/glog/glog.go:879   1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0x8c2c18 0x8c2af4 0x8c2a2d 0x10057d8 0x8c2f5e 0x8c27cd 0x8c26fd 0x1004972 0x10046a0 0x100473a 0x10064a4 0x42d1fd 0x45c191 #    0x8c2c17    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.WaitFor+0x57                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:310 #    0x8c2af3    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.pollInternal+0x93            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:227 #    0x8c2a2c    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Poll+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:221 #    0x10057d7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1+0x97    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:193 #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 #    0x1004971    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew+0x71    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:192 #    0x100469f    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run+0x9f        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:146 #    0x1004739    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection.RunOrDie+0x59            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:157 #    0x10064a3    main.main+0x6a3                                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/cmd/operator/main.go:115 #    0x42d1fc    runtime.main+0x20c                                                    /usr/local/go/1.9/go/src/runtime/proc.go:185   1 @ 0x42d69c 0x42d78e 0x404b54 0x40483b 0xff0725 0xff552a 0x8c2ecf 0x45c191 #    0xff0724    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).run+0x94                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 #    0xff5529    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.run)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:387 #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71   1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0x8aac9c 0x8b6141 0x6e6234 0x6e5f25 0x6e58c8 0xa1c178 0xe78795 0xe78e4c 0x8c4658 0x45c191 #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                /usr/local/go/1.9/go/src/runtime/sema.go:507 #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                        /usr/local/go/1.9/go/src/sync/cond.go:56 #    0x8aac9b    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.(*pipe).Read+0x9b                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/pipe.go:64 #    0x8b6140    github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2.transportResponseBody.Read+0xb0                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/golang.org/x/net/http2/transport.go:1674 #    0x6e6233    encoding/json.(*Decoder).refill+0x113                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:152 #    0x6e5f24    encoding/json.(*Decoder).readValue+0x274                                            /usr/local/go/1.9/go/src/encoding/json/stream.go:128 #    0x6e58c7    encoding/json.(*Decoder).Decode+0x77                                                /usr/local/go/1.9/go/src/encoding/json/stream.go:57 #    0xa1c177    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read+0x2c7        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:150 #    0xe78794    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode+0x94    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 #    0xe78e4b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch.(*Decoder).Decode+0x7b                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/rest/watch/decoder.go:49 #    0x8c4657    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive+0x137            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:93   1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xfe7736 0xfe51c0 0xff533a 0x8c2f5e 0x8c27cd 0x8c26fd 0xfe4fe7 0xfedf9a 0x45c191 #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507 #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56 #    0xfe7735    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop+0xb5                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:438 #    0xfe51bf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop+0x3f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:150 #    0xff5339    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.processLoop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124 #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 #    0xfe4fe6    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*controller).Run+0x2b6                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:124 #    0xfedf99    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run+0x4c9                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:226   1 @ 0x42d69c 0x42d78e 0x4402c0 0x46ffb0 0xffac75 0xffb6bc 0xffbbec 0xffdddd 0xffdd7b 0xffe92a 0x8c2f5e 0x8c27cd 0x8c26fd 0x45c191 #    0x4402bf    sync.runtime_notifyListWait+0x10f                                                                            /usr/local/go/1.9/go/src/runtime/sema.go:507 #    0x46ffaf    sync.(*Cond).Wait+0x7f                                                                                    /usr/local/go/1.9/go/src/sync/cond.go:56 #    0xffac74    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*Type).Get+0x94                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:124 #    0xffb6bb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).Get+0x3b                                            <autogenerated>:1 #    0xffbbeb    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*rateLimitingType).Get+0x3b                                        <autogenerated>:1 #    0xffdddc    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).processNextItem+0x3c                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:43 #    0xffdd7a    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).runWorker+0x2a                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/sync.go:37 #    0xffe929    github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.(*informer).(github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer.runWorker)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/github.com/coreos/operator-sdk/pkg/sdk/informer/informer.go:88 #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc                                            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                                                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88   1 @ 0x42d69c 0x43d9f8 0x692f95 0x45c191 #    0x692f94    net/http.(*persistConn).writeLoop+0x164    /usr/local/go/1.9/go/src/net/http/transport.go:1759   1 @ 0x42d69c 0x43d9f8 0xfc61b6 0xfda64b 0x45c191 #    0xfc61b5    github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run+0x235    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:294 #    0xfda64a    github.com/couchbase/couchbase-operator/pkg/cluster.New.func1+0x27a        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150   1 @ 0x42d69c 0x43d9f8 0xfebcd8 0xfeb64c 0xff45b3 0x8c2f5e 0x8c27cd 0x8c26fd 0xfea514 0xff52f4 0x8c2e11 0x8c2ecf 0x45c191 #    0xfebcd7    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler+0x287        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:373 #    0xfeb64b    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch+0xffb        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:339 #    0xff45b2    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1+0x32            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:204 #    0x8c2f5d    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1+0x5d            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 #    0x8c27cc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil+0xbc            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 #    0x8c26fc    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until+0x4c                /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 #    0xfea513    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run+0x163            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:203 #    0xff52f3    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm+0x33            /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/controller.go:122 #    0x8c2e10    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1+0x30    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71   1 @ 0x42d69c 0x43d9f8 0xff04d0 0xff556a 0x8c2ecf 0x45c191 #    0xff04cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop+0x18f                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:519 #    0xff5569    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*processorListener).(github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.pop)-fm+0x29    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:388 #    0x8c2ece    github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1+0x4e                                        /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71   1 @ 0x42d69c 0x43d9f8 0xff47d0 0x45c191 #    0xff47cf    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1+0x18f    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/tools/cache/reflector.go:278   1 @ 0x42d69c 0x43d9f8 0xff95ef 0x45c191 #    0xff95ee    github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x39e    /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:208   1 @ 0x7071b2 0x706fb0 0x703b0b 0x70fc48 0x70ff4b 0x67e9d4 0x680080 0x6815f4 0x67d8ad 0x45c191 #    0x7071b1    runtime/pprof.writeRuntimeProfile+0xa1    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:637 #    0x706faf    runtime/pprof.writeGoroutine+0x9f    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:599 #    0x703b0a    runtime/pprof.(*Profile).WriteTo+0x3aa    /usr/local/go/1.9/go/src/runtime/pprof/pprof.go:310 #    0x70fc47    net/http/pprof.handler.ServeHTTP+0x1b7    /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:237 #    0x70ff4a    net/http/pprof.Index+0x1da        /usr/local/go/1.9/go/src/net/http/pprof/pprof.go:248 #    0x67e9d3    net/http.HandlerFunc.ServeHTTP+0x43    /usr/local/go/1.9/go/src/net/http/server.go:1918 #    0x68007f    net/http.(*ServeMux).ServeHTTP+0x12f    /usr/local/go/1.9/go/src/net/http/server.go:2254 #    0x6815f3    net/http.serverHandler.ServeHTTP+0xb3    /usr/local/go/1.9/go/src/net/http/server.go:2619 #    0x67d8ac    net/http.(*conn).serve+0x71c        /usr/local/go/1.9/go/src/net/http/server.go:1801
          simon.murray Simon Murray added a comment -

          Okay so it's 11:21+0100 now, which means in UTC that's 10:21.  You added that comment 21 minutes ago, so 10:00 and the time stamp says the last reconcile was 09:56.  Do you agree it is working and not hanging?

          simon.murray Simon Murray added a comment - Okay so it's 11:21+0100 now, which means in UTC that's 10:21.  You added that comment 21 minutes ago, so 10:00 and the time stamp says the last reconcile was 09:56.  Do you agree it is working and not hanging?

          Yeah. I too agree that there is no problem with the operator now.

          But there is something wrong in the AWS node on which the testing is running.

          I have restarted the test now. Will keep you posted on further updates.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Yeah. I too agree that there is no problem with the operator now. But there is something wrong in the AWS node on which the testing is running. I have restarted the test now. Will keep you posted on further updates.

          Not seeing this problem now. Looks like there was some runtime related issues.

          After rebooting the target node, tests seems to work fine.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Not seeing this problem now. Looks like there was some runtime related issues. After rebooting the target node, tests seems to work fine.

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty