Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2212

[Istio] OpenLDAP container cannot be started

    XMLWordPrintable

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.0
    • .maintenance
    • testing
    • 1

    Description

      Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-gke-pipeline/229/console

      Couchbase server: couchbase/server:6.6.2

      TestCase: TestLDAPCreateAdminUser

      BackTrace:

      07:12:02 === CONT  TestOperator/TestLDAPCreateAdminUser
      07:12:02     util.go:1288: timeout: EOF
      07:12:02     util.go:1289: goroutine 746 [running]:
      07:12:02         runtime/debug.Stack(0x1ee267e, 0x28625a0, 0xc0005bf4a0)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/go/src/runtime/debug/stack.go:24 +0xab
      07:12:02         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc000654a80, 0x35253e0, 0xc000e04d00)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/e2eutil/util.go:1284 +0x34
      07:12:02         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustCheckLDAPServer(0xc000654a80, 0xc0001e8460, 0x2a4fd4c, 0x9, 0xc000dde340, 0x8bb2c97000)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/e2eutil/ldap_util.go:93 +0x27e
      07:12:02         github.com/couchbase/couchbase-operator/test/e2e.setupLDAP(0xc000654a80, 0xc0001e8460, 0x0)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/ldap_test.go:47 +0xb05
      07:12:02         github.com/couchbase/couchbase-operator/test/e2e.TestLDAPCreateAdminUser(0xc000654a80)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/ldap_test.go:69 +0xd8
      07:12:02         testing.tRunner(0xc000654a80, 0x2b1c278)
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/go/src/testing/testing.go:1193 +0x203
      07:12:02         created by testing.(*T).Run
      07:12:02         	/jenkins/workspace/k8s-cbop-gke-pipeline/go/src/testing/testing.go:1238 +0x5d8 

      LDAP error:

      Init new ldap server...
        Backing up /etc/ldap/slapd.d in /var/backups/slapd-2.4.47+dfsg-3~bpo9+1... done.
        Creating initial configuration... done.
        Creating LDAP directory... done.
      invoke-rc.d: could not determine current runlevel
      invoke-rc.d: policy-rc.d denied execution of restart.
      Start OpenLDAP...
      *** /container/run/startup/slapd failed with status 1*** 
      Killing all processes... 

      The same test without Istio passes. 

      (cbopinfo attached)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment -

          Well you should be able to use TLS on plain Istio, it's only when using mTLS at the same time it becomes a problem.  What are we doing to document here?  "Our test framework cannot provision an LDAP server, so don't use this cos we have no idea what's happening".  I'd keep this as an open investigation.

          simon.murray Simon Murray added a comment - Well you should be able to use TLS on plain Istio, it's only when using mTLS at the same time it becomes a problem.  What are we doing to document here?  "Our test framework cannot provision an LDAP server, so don't use this cos we have no idea what's happening".  I'd keep this as an open investigation.
          tommie Tommie McAfee added a comment - - edited

          I think this should become a doc ticket as it isn't an issue with operator.

          The test is not going to work as currently written if Istio doesn't support TLS since the connection from couchbase to ldap occurs over startTLS as noted. 

          If you want to try and resolve this, try running within Isto but setting encryption to None:

          https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/e2espec/ldap.go#L55

           

          The real question is why this fails, which is what we'll need to know to document this.

          Simon Murray is it generally the case that using TLS on top of an Isto network will result in this type of behavior?

           

           

          tommie Tommie McAfee added a comment - - edited I think this should become a doc ticket as it isn't an issue with operator. The test is not going to work as currently written if Istio doesn't support TLS since the connection from couchbase to ldap occurs over startTLS as noted.  If you want to try and resolve this, try running within Isto but setting encryption to None: https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/e2espec/ldap.go#L55   The real question is why this fails, which is what we'll need to know to document this. Simon Murray is it generally the case that using TLS on top of an Isto network will result in this type of behavior?    
          prateek.kumar Prateek Kumar (Inactive) added a comment - - edited

          Simon Murray pointed out the base image used for OpenLDAP init container is busybox. ( https://github.com/osixia/docker-light-baseimage/blob/master/image/Dockerfile#L1)

          As advised above, I tried running it in isolation outside operator env. with Istio.

          The command used to run the container is "./tool/run" , This was changed to sleep for few minutes so that the start command can be run manually on the container to check for status of LDAP server.

          A pod was created which brought up both Istio and modified LDAP container. The command was then run manually inside the LDAP container and the server started without any issue.

          Apart from this step the image tags used in the test framework was updated to latest for openLDAP and openLDAPInit image but that didn't make the test pass as well.

          Just to paraphrase , this test has been failing with Istio where we don't support TLS. However , LDAP seems to utilise TLS while creating server and checking for status.

          Another modification tested was creating the LDAP server without TLS but the Couchbase pod failed to connect to LDAP server with error:  Failed to use StartTLS extension 

          prateek.kumar Prateek Kumar (Inactive) added a comment - - edited Simon Murray pointed out the base image used for OpenLDAP init container is busybox. (  https://github.com/osixia/docker-light-baseimage/blob/master/image/Dockerfile#L1 ) As advised above, I tried running it in isolation outside operator env. with Istio. The command used to run the container is "./tool/run" , This was changed to sleep for few minutes so that the start command can be run manually on the container to check for status of LDAP server. A pod was created which brought up both Istio and modified LDAP container. The command was then run manually inside the LDAP container and the server started without any issue. Apart from this step the image tags used in the test framework was updated to latest for openLDAP and openLDAPInit image but that didn't make the test pass as well. Just to paraphrase , this test has been failing with Istio where we don't support TLS. However , LDAP seems to utilise TLS while creating server and checking for status. Another modification tested was creating the LDAP server without TLS but the Couchbase pod failed to connect to LDAP server with error:   Failed to use StartTLS extension  
          simon.murray Simon Murray added a comment -

          We'll need some more logs off the LDAP container. Run it in isolation and find out what's gone on from /var/logs. It may potentially involve TLS as you've not specified the mTLS mode.

          simon.murray Simon Murray added a comment - We'll need some more logs off the LDAP container. Run it in isolation and find out what's gone on from /var/logs. It may potentially involve TLS as you've not specified the mTLS mode.

          People

            tommie Tommie McAfee
            prateek.kumar Prateek Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty