Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3308

CNG containers fails to start with a different issue on each pod

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • None
    • 2.6.0
    • operator
    • None
    • Pre-Christmas Completions, Christmas and Beyond, 1 - 2024 A New Dawn
    • 0

    Description

      We have a couchbase cluster with 3 replicas using CNG. In each pod, CNG fails for different reasons according to the logs. 

      Log summary:

      1. In pod 0000 we see protostellar starting and then receiving SIGTERM for failing liveness probe.
      2. In pod 0001 CNG fails with `failed to reconfigure cbauth` then receiving SIGTERM for failing liveness probe (HTTP probe failed with statuscode: 500)
      3. In pod 0002 CNG starts and then receiving SIGTERM for failing liveness probe (
        HTTP probe failed with statuscode: 404)

      I have attached logs for all three containers as they display different issues during startup before being terminated. I also attached operator logs, cluster describe result and a truncated part of k8s events.

       

      Looking at the operator logs, I see a lot of `unknown field \"spec.servers[0].pod.spec.containers\"`, but last time I checked this is a message should not result in failure.
      General details:

      EKS - 1.25

      Server: 7.2.4-7045

      Operator: 2.6.0-158

      CNG: 0.2.0-142

       

      Probably one of the issue is related to K8S-3288

      Attachments

        1. cao.log
          7.67 MB
        2. cng-0000.log
          13 kB
        3. cng-0001.log
          16 kB
        4. cng-0002.log
          17 kB
        5. describe-pod-0000.log
          9 kB
        6. describe-pod-0001.log
          9 kB
        7. describe-pod-0002.log
          9 kB
        8. events.log
          17 kB
        9. ncg-0.log
          10 kB
        10. ncg-1.log
          10 kB
        11. ncg-2.log
          10 kB
        12. operator.log
          19 kB
        13. pod0000_describe.log
          11 kB
        14. pod0001_describe.log
          11 kB
        15. pod0002_describe.log
          11 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            justin.ashworth Justin Ashworth
            salim.salim Salim Salim
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty