Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.2.0
-
1
Description
Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-gke-pipeline/229/console
Couchbase server: couchbase/server:6.6.2
TestCase: TestLDAPCreateAdminUser
BackTrace:
07:12:02 === CONT TestOperator/TestLDAPCreateAdminUser |
07:12:02 util.go:1288: timeout: EOF |
07:12:02 util.go:1289: goroutine 746 [running]: |
07:12:02 runtime/debug.Stack(0x1ee267e, 0x28625a0, 0xc0005bf4a0) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/go/src/runtime/debug/stack.go:24 +0xab |
07:12:02 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc000654a80, 0x35253e0, 0xc000e04d00) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/e2eutil/util.go:1284 +0x34 |
07:12:02 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustCheckLDAPServer(0xc000654a80, 0xc0001e8460, 0x2a4fd4c, 0x9, 0xc000dde340, 0x8bb2c97000) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/e2eutil/ldap_util.go:93 +0x27e |
07:12:02 github.com/couchbase/couchbase-operator/test/e2e.setupLDAP(0xc000654a80, 0xc0001e8460, 0x0) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/ldap_test.go:47 +0xb05 |
07:12:02 github.com/couchbase/couchbase-operator/test/e2e.TestLDAPCreateAdminUser(0xc000654a80) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/test/e2e/ldap_test.go:69 +0xd8 |
07:12:02 testing.tRunner(0xc000654a80, 0x2b1c278) |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/go/src/testing/testing.go:1193 +0x203 |
07:12:02 created by testing.(*T).Run |
07:12:02 /jenkins/workspace/k8s-cbop-gke-pipeline/go/src/testing/testing.go:1238 +0x5d8 |
LDAP error:
Init new ldap server... |
Backing up /etc/ldap/slapd.d in /var/backups/slapd-2.4.47+dfsg-3~bpo9+1... done. |
Creating initial configuration... done.
|
Creating LDAP directory... done.
|
invoke-rc.d: could not determine current runlevel
|
invoke-rc.d: policy-rc.d denied execution of restart.
|
Start OpenLDAP...
|
*** /container/run/startup/slapd failed with status 1*** |
Killing all processes...
|
The same test without Istio passes.
(cbopinfo attached)
Attachments
Gerrit Reviews
Activity
I think this should become a doc ticket as it isn't an issue with operator.
The test is not going to work as currently written if Istio doesn't support TLS since the connection from couchbase to ldap occurs over startTLS as noted.
If you want to try and resolve this, try running within Isto but setting encryption to None:
https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/e2espec/ldap.go#L55
The real question is why this fails, which is what we'll need to know to document this.
Simon Murray is it generally the case that using TLS on top of an Isto network will result in this type of behavior?
Simon Murray pointed out the base image used for OpenLDAP init container is busybox. ( https://github.com/osixia/docker-light-baseimage/blob/master/image/Dockerfile#L1)
As advised above, I tried running it in isolation outside operator env. with Istio.
The command used to run the container is "./tool/run" , This was changed to sleep for few minutes so that the start command can be run manually on the container to check for status of LDAP server.
A pod was created which brought up both Istio and modified LDAP container. The command was then run manually inside the LDAP container and the server started without any issue.
Apart from this step the image tags used in the test framework was updated to latest for openLDAP and openLDAPInit image but that didn't make the test pass as well.
Just to paraphrase , this test has been failing with Istio where we don't support TLS. However , LDAP seems to utilise TLS while creating server and checking for status.
Another modification tested was creating the LDAP server without TLS but the Couchbase pod failed to connect to LDAP server with error: Failed to use StartTLS extension
We'll need some more logs off the LDAP container. Run it in isolation and find out what's gone on from /var/logs. It may potentially involve TLS as you've not specified the mTLS mode.
Well you should be able to use TLS on plain Istio, it's only when using mTLS at the same time it becomes a problem. What are we doing to document here? "Our test framework cannot provision an LDAP server, so don't use this cos we have no idea what's happening". I'd keep this as an open investigation.