Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-612

Pod Creation Failure Should Report The Phase That Failed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 1.1.0
    • operator
    • None
    • Azure AKS

    Description

      Cannot setup TLS with couchbase operator

      time="2018-10-02T03:43:29Z" level=info msg="Setting up secure client for operator communication with the cluster" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:43:29Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:43:29Z" level=info msg="Creating headless service for data nodes" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:43:29Z" level=info msg="Creating NodePort UI service (cb-op-aks-cluster-mini-ui) for data nodes" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:43:30Z" level=info msg="Creating a pod (cb-op-aks-cluster-mini-0000) running Couchbase enterprise-5.5.1" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:47:41Z" level=info msg="deleted pod (cb-op-aks-cluster-mini-0000)" cluster-name=cb-op-aks-cluster-mini module=cluster

      time="2018-10-02T03:47:41Z" level=error msg="Cluster setup failed: context deadline exceeded" cluster-name=cb-op-aks-cluster-mini module=cluster
      time="2018-10-02T03:47:41Z" level=warning msg="Fail to handle event: ignore failed cluster (cb-op-aks-cluster-mini). Please delete its CR"

       pods gets deleted when operator gives up on the cluster

      pod stays in init state for long time

      $ kubectl get pods
      NAME READY STATUS RESTARTS AGE cb-op-aks-cluster-mini-0000 0/1 Init:0/1 0 6s
      couchbase-operator-6cb7687498-f8wb9 1/1 Running 0 1h
      $ kubectl logs -f cb-op-aks-cluster-mini-0000
      Error from server (BadRequest): container "couchbase-server" in pod "cb-op-aks-cluster-mini-0000" is waiting to start: PodInitializing

       

      $ kubectl logs -f cb-op-aks-cluster-mini-0000
      Starting Couchbase Server – Web UI available at http://<ip>:8091
      and logs available in /opt/couchbase/var/lib/couchbase/logs
      chown: changing ownership of 'var/lib/couchbase/inbox/..data': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox/pkey.key': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox/chain.pem': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox/..2018_10_02_03_43_47.616069532/pkey.key': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox/..2018_10_02_03_43_47.616069532/chain.pem': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox/..2018_10_02_03_43_47.616069532': Read-only file system
      chown: changing ownership of 'var/lib/couchbase/inbox': Read-only file system

      $ kubectl logs -f cb-op-aks-cluster-mini-0000
      Error from server (NotFound): pods "cb-op-aks-cluster-mini-0000" not found

      Not sure if this is related to the platform AKS

       

       

      Attachments

        1. ca.crt
          1 kB
        2. chain.pem
          5 kB
        3. pkey.key
          2 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment - - edited

          We are going to create a cluster called "laos" in the namespace "asia"

          Clone EasyRSA

          git clone http://github.com/OpenVPN/easy-rsa


          Cloning into 'easy-rsa'...
          remote: Enumerating objects: 6, done.
          remote: Counting objects: 100% (6/6), done.
          remote: Compressing objects: 100% (6/6), done.
          remote: Total 1004 (delta 0), reused 3 (delta 0), pack-reused 998
          Receiving objects: 100% (1004/1004), 851.59 KiB | 1.00 MiB/s, done.
          Resolving deltas: 100% (440/440), done.
          cd easy-rsa/easyrsa3/

          Initialize your PKI

          ./easyrsa init-pki

          init-pki complete; you may now create a CA or requests.
          Your newly created PKI dir is: /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki

          Create your CA

          ./easyrsa build-ca

          Using SSL: openssl OpenSSL 1.1.0g 2 Nov 2017

          Enter New CA Key Passphrase:
          Re-Enter New CA Key Passphrase:
          Generating RSA private key, 2048 bit long modulus
          ................................................................................................+++
          .........+++
          e is 65537 (0x010001)
          You are about to be asked to enter information that will be incorporated
          into your certificate request.
          What you are about to enter is what is called a Distinguished Name or a DN.
          There are quite a few fields but you can leave some blank
          For some fields there will be a default value,
          If you enter '.', the field will be left blank.
          -----
          Common Name (eg: your user, host, or server name) [Easy-RSA CA]:Asia CACA creation complete and you may now import and sign cert requests.
          Your new CA certificate file for publishing is at:
          /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/ca.crt

          Create your server certificate specific to the cluster and namespace.

          ./easyrsa --subject-alt-name=DNS:.laos.asia.svc build-server-full couchbase-server nopass

          Using SSL: openssl OpenSSL 1.1.0g 2 Nov 2017
          Generating a 2048 bit RSA private key
          ........................................................+++
          .......................................+++
          writing new private key to '/home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/private/couchbase-server.key.RafAqhXzTG'
          -----
          Using configuration from /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/safessl-easyrsa.cnf
          Enter pass phrase for /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/private/ca.key:
          Can't open /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/index.txt.attr for reading, No such file or directory
          139845823746496:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:74:fopen('/home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/index.txt.attr','r')
          139845823746496:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:81:
          Check that the request matches the signature
          Signature ok
          The Subject's Distinguished Name is as follows
          commonName :ASN.1 12:'couchbase-server'
          Certificate is to be certified until Oct 6 08:40:20 2021 GMT (1080 days)Write out database with 1 new entries
          Data Base Updated

          Reformat the private key so NS server does't complain

          openssl rsa -in pkey.key -out pkey.key.der -outform DER
          writing RSA key
          openssl rsa -in pkey.key.der -inform DER -out pkey.key -outform PEM
          writing RSA key

          Create your secrets in the correct namespace

          cp pki/private/couchbase-server.key pkey.key
          cp pki/issued/couchbase-server.crt chain.pem
          kubectl create secret generic couchbase-server-tls --from-file chain.pem --from-file pkey.key --namespace asia
          secret/couchbase-server-tls created
          kubectl create secret generic couchbase-operator-tls --from-file pki/ca.crt --namespace asia
          secret/couchbase-operator-tls created

          Create a namespace called "asia"

          kubectl create namespace asia
          namespace/asia created

          Create a username and password

          kubectl create -f example/secret.yaml --namespace asia
          secret/cb-example-auth created

          Setup RBAC (not shown)

          Create the cluster

          kubectl create -f example/deployment.yaml --namespace asia
          deployment.extensions/couchbase-operator created
          kubectl create -f example/tls/couchbase-cluster.yaml --namespace asia
          couchbasecluster.couchbase.com/cb-example created

          Watch as it comes up

          kubectl logs -f deployment/couchbase-operator --namespace asia
          time="2018-10-22T08:53:31Z" level=info msg="couchbase-operator v1.1.0 (k8s_312_pod_disruption 2f097dc7286d6fa6abcf84af575794808085c733)" module=main
          time="2018-10-22T08:53:31Z" level=info msg="Obtaining resource lock" module=main
          time="2018-10-22T08:53:31Z" level=info msg="Starting event recorder" module=main
          time="2018-10-22T08:53:31Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
          time="2018-10-22T08:53:49Z" level=info msg="I'm the leader, attempt to start the operator" module=main
          time="2018-10-22T08:53:49Z" level=info msg="Creating the couchbase-operator controller" module=main
          time="2018-10-22T08:53:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"asia\", Name:\"couchbase-operator\", UID:\"35216050-d5d7-11e8-a74a-080027b049c1\", APIVersion:\"v1\", ResourceVersion:\"577577\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-766f64499c-sdphk became leader" module=event_recorder
          time="2018-10-22T08:53:49Z" level=info msg="CRD initialized, listening for events..." module=controller
          time="2018-10-22T08:53:49Z" level=info msg="starting couchbaseclusters controller"
          time="2018-10-22T08:53:49Z" level=info msg="Watching new cluster" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Janitor process starting" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Setting up secure client for operator communication with the cluster" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Creating headless service for data nodes" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Creating NodePort UI service (laos-ui) for data nodes" cluster-name=laos module=cluster
          time="2018-10-22T08:53:49Z" level=info msg="Creating a pod (laos-0000) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster
          time="2018-10-22T08:54:02Z" level=info msg="Operator added member (laos-0000) to manage" cluster-name=laos module=cluster
          time="2018-10-22T08:54:02Z" level=info msg="Initializing the first node in the cluster" cluster-name=laos module=cluster
          time="2018-10-22T08:54:02Z" level=info msg="start running..." cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="server config all_services: laos-0000" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="Cluster status: balanced" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="Node status:" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="┌───────────┬──────────────┬────────────────┐" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="│ Server │ Class │ Status │" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="├───────────┼──────────────┼────────────────┤" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="│ laos-0000 │ all_services │ managed+active │" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info msg="└───────────┴──────────────┴────────────────┘" cluster-name=laos module=cluster
          time="2018-10-22T08:54:10Z" level=info cluster-name=laos module=cluster
          time="2018-10-22T08:54:11Z" level=info msg="Creating a pod (laos-0001) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster
          time="2018-10-22T08:54:26Z" level=info msg="added member (laos-0001)" cluster-name=laos module=cluster
          time="2018-10-22T08:54:26Z" level=info msg="Creating a pod (laos-0002) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster
          time="2018-10-22T08:54:40Z" level=info msg="added member (laos-0002)" cluster-name=laos module=cluster
          time="2018-10-22T08:54:44Z" level=info msg="Rebalance progress: 0.000000" cluster-name=laos module=cluster
          time="2018-10-22T08:54:52Z" level=info msg="reconcile finished" cluster-name=laos module=cluster
          time="2018-10-22T08:54:57Z" level=info msg="Created bucket default" cluster-name=laos module=cluster

           

          Easy!

          simon.murray Simon Murray added a comment - - edited We are going to create a cluster called "laos" in the namespace "asia" Clone EasyRSA git clone http://github.com/OpenVPN/easy-rsa Cloning into 'easy-rsa'... remote: Enumerating objects: 6, done. remote: Counting objects: 100% (6/6), done. remote: Compressing objects: 100% (6/6), done. remote: Total 1004 (delta 0), reused 3 (delta 0), pack-reused 998 Receiving objects: 100% (1004/1004), 851.59 KiB | 1.00 MiB/s, done. Resolving deltas: 100% (440/440), done. cd easy-rsa/easyrsa3/ Initialize your PKI ./easyrsa init-pki init-pki complete; you may now create a CA or requests. Your newly created PKI dir is: /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki Create your CA ./easyrsa build-ca Using SSL: openssl OpenSSL 1.1.0g 2 Nov 2017 Enter New CA Key Passphrase: Re-Enter New CA Key Passphrase: Generating RSA private key, 2048 bit long modulus ................................................................................................+++ .........+++ e is 65537 (0x010001) You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Common Name (eg: your user, host, or server name) [Easy-RSA CA] :Asia CA CA creation complete and you may now import and sign cert requests. Your new CA certificate file for publishing is at: /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/ca.crt Create your server certificate specific to the cluster and namespace. ./easyrsa --subject-alt-name=DNS:.laos.asia.svc build-server-full couchbase-server nopass Using SSL: openssl OpenSSL 1.1.0g 2 Nov 2017 Generating a 2048 bit RSA private key ........................................................+++ .......................................+++ writing new private key to '/home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/private/couchbase-server.key.RafAqhXzTG' ----- Using configuration from /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/safessl-easyrsa.cnf Enter pass phrase for /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/private/ca.key: Can't open /home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/index.txt.attr for reading, No such file or directory 139845823746496:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:74:fopen('/home/simon/go/src/github.com/couchbase/couchbase-operator/easy-rsa/easyrsa3/pki/index.txt.attr','r') 139845823746496:error:2006D080:BIO routines:BIO_new_ file:no such file:../crypto/bio/bss_file.c:81: Check that the request matches the signature Signature ok The Subject's Distinguished Name is as follows commonName :ASN.1 12:'couchbase-server' Certificate is to be certified until Oct 6 08:40:20 2021 GMT (1080 days) Write out database with 1 new entries Data Base Updated Reformat the private key so NS server does't complain openssl rsa -in pkey.key -out pkey.key.der -outform DER writing RSA key openssl rsa -in pkey.key.der -inform DER -out pkey.key -outform PEM writing RSA key Create your secrets in the correct namespace cp pki/private/couchbase-server.key pkey.key cp pki/issued/couchbase-server.crt chain.pem kubectl create secret generic couchbase-server-tls --from-file chain.pem --from-file pkey.key --namespace asia secret/couchbase-server-tls created kubectl create secret generic couchbase-operator-tls --from-file pki/ca.crt --namespace asia secret/couchbase-operator-tls created Create a namespace called "asia" kubectl create namespace asia namespace/asia created Create a username and password kubectl create -f example/secret.yaml --namespace asia secret/cb-example-auth created Setup RBAC (not shown) Create the cluster kubectl create -f example/deployment.yaml --namespace asia deployment.extensions/couchbase-operator created kubectl create -f example/tls/couchbase-cluster.yaml --namespace asia couchbasecluster.couchbase.com/cb-example created Watch as it comes up kubectl logs -f deployment/couchbase-operator --namespace asia time="2018-10-22T08:53:31Z" level=info msg="couchbase-operator v1.1.0 (k8s_312_pod_disruption 2f097dc7286d6fa6abcf84af575794808085c733)" module=main time="2018-10-22T08:53:31Z" level=info msg="Obtaining resource lock" module=main time="2018-10-22T08:53:31Z" level=info msg="Starting event recorder" module=main time="2018-10-22T08:53:31Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main time="2018-10-22T08:53:49Z" level=info msg="I'm the leader, attempt to start the operator" module=main time="2018-10-22T08:53:49Z" level=info msg="Creating the couchbase-operator controller" module=main time="2018-10-22T08:53:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"asia\", Name:\"couchbase-operator\", UID:\"35216050-d5d7-11e8-a74a-080027b049c1\", APIVersion:\"v1\", ResourceVersion:\"577577\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-766f64499c-sdphk became leader" module=event_recorder time="2018-10-22T08:53:49Z" level=info msg="CRD initialized, listening for events..." module=controller time="2018-10-22T08:53:49Z" level=info msg="starting couchbaseclusters controller" time="2018-10-22T08:53:49Z" level=info msg="Watching new cluster" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Janitor process starting" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Setting up secure client for operator communication with the cluster" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Creating headless service for data nodes" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Creating NodePort UI service (laos-ui) for data nodes" cluster-name=laos module=cluster time="2018-10-22T08:53:49Z" level=info msg="Creating a pod (laos-0000) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster time="2018-10-22T08:54:02Z" level=info msg="Operator added member (laos-0000) to manage" cluster-name=laos module=cluster time="2018-10-22T08:54:02Z" level=info msg="Initializing the first node in the cluster" cluster-name=laos module=cluster time="2018-10-22T08:54:02Z" level=info msg="start running..." cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="server config all_services: laos-0000" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="Cluster status: balanced" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="Node status:" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="┌───────────┬──────────────┬────────────────┐" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="│ Server │ Class │ Status │" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="├───────────┼──────────────┼────────────────┤" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="│ laos-0000 │ all_services │ managed+active │" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info msg="└───────────┴──────────────┴────────────────┘" cluster-name=laos module=cluster time="2018-10-22T08:54:10Z" level=info cluster-name=laos module=cluster time="2018-10-22T08:54:11Z" level=info msg="Creating a pod (laos-0001) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster time="2018-10-22T08:54:26Z" level=info msg="added member (laos-0001)" cluster-name=laos module=cluster time="2018-10-22T08:54:26Z" level=info msg="Creating a pod (laos-0002) running Couchbase enterprise-5.5.1" cluster-name=laos module=cluster time="2018-10-22T08:54:40Z" level=info msg="added member (laos-0002)" cluster-name=laos module=cluster time="2018-10-22T08:54:44Z" level=info msg="Rebalance progress: 0.000000" cluster-name=laos module=cluster time="2018-10-22T08:54:52Z" level=info msg="reconcile finished" cluster-name=laos module=cluster time="2018-10-22T08:54:57Z" level=info msg="Created bucket default" cluster-name=laos module=cluster   Easy!
          simon.murray Simon Murray added a comment -

          Reopening so I can  edit... stupid jira

          simon.murray Simon Murray added a comment - Reopening so I can  edit... stupid jira
          simon.murray Simon Murray added a comment -

          sigh stop reformatting!!!!

          simon.murray Simon Murray added a comment - sigh stop reformatting!!!!
          simon.murray Simon Murray added a comment -

          I give up, live with it

          simon.murray Simon Murray added a comment - I give up, live with it

          Thanks Simon for the detailed. Steps to reformat private keys should come after cp command to create pkey.key.

          I could setup CB k8s cluster running self-signed cert and verified the cert contents that matches the SAN from downloaded cert.

          ram.dhakne Ram Dhakne (Inactive) added a comment - Thanks Simon for the detailed. Steps to reformat private keys should come after cp command to create pkey.key. I could setup CB k8s cluster running self-signed cert and verified the cert contents that matches the SAN from downloaded cert.

          People

            daniel.ma Daniel Ma (Inactive)
            ram.dhakne Ram Dhakne (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty