Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-573

Cannot create two couchbase clusters with one deployment of operator

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Done
    • 1.0.0
    • 1.1.0
    • kubernetes
    • None
    • Azure AKS

    Description

      I have a Couchbase cluster running on Azure AKS

      Operator version: 1.0

      K8s version: v1.11.2

      I have a couchbase cluster with name: cb-op-aks-demo

      $ kubectl get pods
      NAME READY STATUS RESTARTS AGE
      cb-op-aks-demo-0000 1/1 Running 0 6d
      cb-op-aks-demo-0002 1/1 Running 0 6d
      cb-op-aks-demo-0003 1/1 Running 0 6d
      cb-op-aks-demo-0004 1/1 Running 0 6d
      cb-op-aks-demo-0005 1/1 Running 0 6d
      cb-op-aks-demo-0006 1/1 Running 0 6d
      couchbase-operator-6cb7687498-zfzq5 1/1 Running 3 6d

      Its all working fine.

      Trying to create one more couchbase cluster fails

      From operator logs

       

      E0906 20:48:18.425482 1 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 1531; INTERNAL_ERROR
      time="2018-09-06T21:00:44Z" level=info msg="Watching new cluster" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:00:45Z" level=info msg="Setting up client for operator communication with the cluster" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:00:45Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:00:45Z" level=info msg="Creating headless service for data nodes" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:00:46Z" level=info msg="Creating NodePort UI service (cb-op-aks-xdcr-demo-ui) for data nodes" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:00:47Z" level=info msg="Creating a pod (cb-op-aks-xdcr-demo-0000) running Couchbase enterprise-5.5.1" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:06:07Z" level=info msg="deleted pod (cb-op-aks-xdcr-demo-0000)" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:06:07Z" level=error msg="Cluster setup failed: context deadline exceeded" cluster-name=cb-op-aks-xdcr-demo module=cluster
      time="2018-09-06T21:06:08Z" level=warning msg="Fail to handle event: ignore failed cluster (cb-op-aks-xdcr-demo). Please delete its CR"

       

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Can you add the logs from cbopinfo?

          https://docs.couchbase.com/operator/1.0/cbopinfo.html

           

          Also, it looks like the creation of the second cluster failed likely due to the initial pod not being able to be created. We can check to see why that happened, but there doesn't appear to be any issue with running multiple clusters in the same namespace.

          mikew Mike Wiederhold [X] (Inactive) added a comment - Can you add the logs from cbopinfo? https://docs.couchbase.com/operator/1.0/cbopinfo.html   Also, it looks like the creation of the second cluster failed likely due to the initial pod not being able to be created. We can check to see why that happened, but there doesn't appear to be any issue with running multiple clusters in the same namespace.

          Added cbopinfo diags bundle.

          ram.dhakne Ram Dhakne (Inactive) added a comment - Added cbopinfo diags bundle.

          Ram,

          AKS is very slow allocating persistent volumes sometimes. Our timeout for pod creation is 5 minutes which in my opinion is very long. I would suggest contacting Microsoft support to find out why the volumes are taking so long to provision.

          mikew Mike Wiederhold [X] (Inactive) added a comment - Ram, AKS is very slow allocating persistent volumes sometimes. Our timeout for pod creation is 5 minutes which in my opinion is very long. I would suggest contacting Microsoft support to find out why the volumes are taking so long to provision.

          People

            mikew Mike Wiederhold [X] (Inactive)
            ram.dhakne Ram Dhakne (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty