Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-510

Operator panics while removing nodes from the server config

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • operator

    Description

       

      Test case: TestBasicMDSScaling

      couchbase-operator]$ kubectl get pod
      NAME                                  READY     STATUS    RESTARTS   AGE
      couchbase-operator-5bdf548959-fn2xv   1/1       Running   3          10m
      test-couchbase-c28v9-0000             1/1       Running   0          10m
      Ashwins-MacBook-Pro:couchbase-operator]${noformat}
      *Operator log prints:*
       
       
      

      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xfc7a48]
      goroutine 88 [running]:
      github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1.(*ServerConfig).GetVolumeMounts(...)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1/cluster.go:390
      github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).removePod(0xc420086d80, 0xc420126a00, 0x19, 0xc4208efb40, 0xc4208efb30)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:467 +0x328
      github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).destroyMember(0xc420086d80, 0xc420126a00, 0x19, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:211 +0x43
      github.com/couchbase/couchbase-operator/pkg/cluster.handleDeadMembers(0xc420125500, 0xc420086d80, 0x12f0590, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:556 +0xe9
      github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step(0xc420125500, 0xc420086d80, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:93 +0x6b
      github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec(0xc420125500, 0xc420086d80, 0xc420210a01, 0xc420125500)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:108 +0x47
      github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers(0xc420086d80, 0xc420125500, 0x0, 0xc420125440)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:124 +0x35
      github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile(0xc420086d80, 0xc42028bdc0, 0x3, 0x4, 0x0, 0x0)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:80 +0x42d
      github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run(0xc420086d80)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:350 +0x723
      github.com/couchbase/couchbase-operator/pkg/cluster.New.func1(0xc420086d80)
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150 +0x27b
      created by github.com/couchbase/couchbase-operator/pkg/cluster.New
          /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:137 +0x86a

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Observing similar issue while running TestRemoveLastDataService

           

           

          time="2018-08-04T09:28:29Z" level=info msg="Member test-couchbase-5xdrx-0001 is no longer part of any server config, removing" cluster-name=test-couchbase-5xdrx module=cluster
          time="2018-08-04T09:28:32Z" level=info msg="Rebalance progress: 0.000000" cluster-name=test-couchbase-5xdrx module=cluster
          time="2018-08-04T09:28:41Z" level=info msg="Finish reconciling" cluster-name=test-couchbase-5xdrx module=cluster
          panic: runtime error: invalid memory address or nil pointer dereference
          [signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xfc7a48]
          goroutine 2278 [running]:
          github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1.(*ServerConfig).GetVolumeMounts(...)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1/cluster.go:390
          github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).removePod(0xc420208b40, 0xc4209172a0, 0x19, 0xc4208d3b40, 0xc4208d3b30)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:467 +0x328
          github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).destroyMember(0xc420208b40, 0xc4209172a0, 0x19, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:211 +0x43
          github.com/couchbase/couchbase-operator/pkg/cluster.handleDeadMembers(0xc4206f9c20, 0xc420208b40, 0x12f0590, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:556 +0xe9
          github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step(0xc4206f9c20, 0xc420208b40, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:93 +0x6b
          github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec(0xc4206f9c20, 0xc420208b40, 0xc42040b601, 0xc4206f9c20)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:108 +0x47
          github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers(0xc420208b40, 0xc4206f9c20, 0x0, 0xc4206f9bf0)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:124 +0x35
          github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile(0xc420208b40, 0xc4202d7290, 0x2, 0x2, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:80 +0x42d
          github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run(0xc420208b40)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:350 +0x723
          github.com/couchbase/couchbase-operator/pkg/cluster.New.func1(0xc420208b40)
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150 +0x27b
          created by github.com/couchbase/couchbase-operator/pkg/cluster.New
              /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:137 +0x86a
          Ashwins-MacBook-Pro:couchbase-operator]${noformat}

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Observing similar issue while running TestRemoveLastDataService     time="2018-08-04T09:28:29Z" level=info msg="Member test-couchbase-5xdrx-0001 is no longer part of any server config, removing" cluster-name=test-couchbase-5xdrx module=cluster time="2018-08-04T09:28:32Z" level=info msg="Rebalance progress: 0.000000" cluster-name=test-couchbase-5xdrx module=cluster time="2018-08-04T09:28:41Z" level=info msg="Finish reconciling" cluster-name=test-couchbase-5xdrx module=cluster panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xfc7a48] goroutine 2278 [running] : github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1.(*ServerConfig).GetVolumeMounts(...)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/apis/couchbase/v1/cluster.go:390 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).removePod(0xc420208b40, 0xc4209172a0, 0x19, 0xc4208d3b40, 0xc4208d3b30)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:467 +0x328 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).destroyMember(0xc420208b40, 0xc4209172a0, 0x19, 0x0, 0x0)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:211 +0x43 github.com/couchbase/couchbase-operator/pkg/cluster.handleDeadMembers(0xc4206f9c20, 0xc420208b40, 0x12f0590, 0x0)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:556 +0xe9 github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step(0xc4206f9c20, 0xc420208b40, 0x0, 0x0)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:93 +0x6b github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec(0xc4206f9c20, 0xc420208b40, 0xc42040b601, 0xc4206f9c20)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:108 +0x47 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers(0xc420208b40, 0xc4206f9c20, 0x0, 0xc4206f9bf0)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:124 +0x35 github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile(0xc420208b40, 0xc4202d7290, 0x2, 0x2, 0x0, 0x0)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:80 +0x42d github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).run(0xc420208b40)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:350 +0x723 github.com/couchbase/couchbase-operator/pkg/cluster.New.func1(0xc420208b40)     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:150 +0x27b created by github.com/couchbase/couchbase-operator/pkg/cluster.New     /home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:137 +0x86a Ashwins-MacBook-Pro:couchbase-operator]${noformat}

          I'm assuming serverConfig here is 'nil', I'm attempting to verify using the e2e test

          func (sc *ServerConfig) GetVolumeMounts() *VolumeMounts {
           
            if sc.Pod != nil {

          tommie Tommie McAfee added a comment - I'm assuming serverConfig here is 'nil', I'm attempting to verify using the e2e test func (sc *ServerConfig) GetVolumeMounts() *VolumeMounts {     if sc.Pod != nil {

          Operator panic is fixed and test case is working fine.

          Verified using docker image: couchbase/couchbase-operator-internal:1.0.0-376

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Operator panic is fixed and test case is working fine. Verified using docker image: couchbase/couchbase-operator-internal:1.0.0-376

          People

            tommie Tommie McAfee
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty