Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-175

collect-info job not working on AWS Openshift environments

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 0.7.0
    • None
    • operator
    • None

    Description

      Objective: Collecting logs from the Couchbase Cluster using the steps in the guide.

      Tried using the step below:

      kubectl create -f https://packages.couchbase.com/kubernetes/beta/couchbase-cli-collect-logs.yaml

      The state of the container is always shown as Container Creating. It appears that a container is tried to be created and then the container creation fails and then another new container is being created.

      [root@ip-172-31-17-218 ~]# kubectl get pods
      ]NAME                                  READY     STATUS              RESTARTS   AGE
      cb-example-0000                       1/1       Running             0          33m
      cb-example-0001                       1/1       Running             0          33m
      cb-example-0002                       1/1       Running             0          32m
      collect-info-x4clv                    0/1       ContainerCreating   0          1s

      [root@ip-172-31-17-218 ~]# kubectl get pods
      NAME                                  READY     STATUS              RESTARTS   AGE
      cb-example-0000                       1/1       Running             0          34m
      cb-example-0001                       1/1       Running             0          34m
      cb-example-0002                       1/1       Running             0          33m
      collect-info-7bjjm                    0/1       ContainerCreating   0          2s

      I dont see any logs generated in any of the pods. I tried logging into the docker containers corresponding to the pods "cb-example-0000", "cb-example-0001" and "cb-example-0002" and checked in the location: /opt/couchbase/var/lib/couchbase/tmp/

      [root@ip-172-31-17-218 ~]# kubectl describe pod collect-info-7bjjm
      Name:        collect-info-7bjjm
      Namespace:    myproject
      Node:        ip-172-31-17-231.us-west-1.compute.internal/172.31.17.231
      Start Time:    Sat, 03 Feb 2018 17:56:55 +0000
      Labels:        controller-uid=a3f6e5bf-0906-11e8-b9ef-02e60b2e7714
              job-name=collect-info
      Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"Job","namespace":"myproject","name":"collect-info","uid":"a3f6e5bf-0906-11e8-b9ef-02e60b2e7714","apiVersio...
              openshift.io/scc=anyuid
      Status:        Failed
      IP:        10.131.1.39
      Created By:    Job/collect-info
      Controlled By:    Job/collect-info
      Containers:
        collect-info:
          Container ID:    docker://16f24b9aba5f081158ca0bfe8d50ee44b6e0ee579875717ed906e9e25c361674
          Image:        sequoiatools/couchbase-cli
          Image ID:        docker-pullable://docker.io/sequoiatools/couchbase-cli@sha256:113aba533572de856f8925c106d5f2efb0380517087c3dba1933308f0a5de862
          Port:        <none>
          Command:
            /bin/sh
            -c
            /couchbase-cli-secure collect-logs-start -c cb-example-0000.cb-example.default.svc -u {auth.admin.username} -p {auth.admin.password} --all-nodes
          State:        Terminated
            Reason:        Error
            Exit Code:    1
            Started:        Sat, 03 Feb 2018 17:56:57 +0000
            Finished:        Sat, 03 Feb 2018 17:56:57 +0000
          Ready:        False
          Restart Count:    0
          Environment:    <none>
          Mounts:
            /auth/admin from cb-admin (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from default-token-jgqb4 (ro)
      Conditions:
        Type        Status
        Initialized     True
        Ready     False
        PodScheduled     True
      Volumes:
        cb-admin:
          Type:    Secret (a volume populated by a Secret)
          SecretName:    cb-example-auth
          Optional:    false
        default-token-jgqb4:
          Type:    Secret (a volume populated by a Secret)
          SecretName:    default-token-jgqb4
          Optional:    false
      QoS Class:    BestEffort
      Node-Selectors:    <none>
      Tolerations:    <none>
      Events:
        FirstSeen    LastSeen    Count    From                            SubObjectPath            Type        Reason            Message
        ---------    --------    -----    ----                            -------------            --------    ------            -------
        9s        9s        1    default-scheduler                        Normal        Scheduled        Successfully assigned collect-info-7bjjm to ip-172-31-17-231.us-west-1.compute.internal
        8s        8s        1    kubelet, ip-172-31-17-231.us-west-1.compute.internal        Normal        SuccessfulMountVolume    MountVolume.SetUp succeeded for volume "default-token-jgqb4"
        8s        8s        1    kubelet, ip-172-31-17-231.us-west-1.compute.internal        Normal        SuccessfulMountVolume    MountVolume.SetUp succeeded for volume "cb-admin"
        7s        7s        1    kubelet, ip-172-31-17-231.us-west-1.compute.internal    spec.containers{collect-info}    Normal        Pulled            Container image "sequoiatools/couchbase-cli" already present on machine
        7s        7s        1    kubelet, ip-172-31-17-231.us-west-1.compute.internal    spec.containers{collect-info}    Normal        Created            Created container
        6s        6s        1    kubelet, ip-172-31-17-231.us-west-1.compute.internal    spec.containers{collect-info}    Normal        Started            Started container
      [root@ip-172-31-17-218 ~]#

      The collect job's container logs are as follows:

      [root@ip-172-31-17-231 ~]# docker ps | grep collect
      16ede08f5811        docker.io/sequoiatools/couchbase-cli@sha256:113aba533572de856f8925c106d5f2efb0380517087c3dba1933308f0a5de862              "/bin/sh -c '/couchba"   1 seconds ago       Up Less than a second                       k8s_collect-info_collect-info-qf5sp_myproject_e3a460f2-090b-11e8-b9ef-02e60b2e7714_0
      7739e6f25652        openshift3/ose-pod:v3.7.23                                                                                                "/usr/bin/pod"           2 seconds ago       Up 1 seconds                                k8s_POD_collect-info-qf5sp_myproject_e3a460f2-090b-11e8-b9ef-02e60b2e7714_0
      [root@ip-172-31-17-231 ~]# docker logs 16ede08f5811
      Boolean docker_transition_unconfined is not defined
      ERROR: Cluster is not initialized, use cluster-init to initialize the cluster

      Please let me know what is going wrong and what I am missing.

      Attachments

        For Gerrit Dashboard: K8S-175
        # Subject Branch Project Status CR V

        Activity

          People

            tommie Tommie McAfee (Inactive)
            sindhura.palakodety Sindhura Palakodety (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty