Details
-
Bug
-
Resolution: Not a Bug
-
Minor
-
2.1.0
-
None
-
1
Description
Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-eks-pipeline/33/consoleFull
TestCases Impacted: TestAutoscalerDeleted, TestAutoscaleEnabled, TestAutoscaleEnabled, TestAutoScaleMultiConfigs
Suite: TestP0, TestP1
Platform: EKS
Versions: K8s1.18 (6.0.4/6.6.0)
( there were 7 runs of regression with different Couchbase Server and Server Upgrade versions with k8s 1.17 and 1.18, In case you need to see the series of jobs with logs: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-eks-pipeline/ [30-36] , these tests failed in every of these jobs.)
Error for TestAutoScaleMultiConfigs:
04:51:35 --- FAIL: TestOperator/TestAutoscaleMultiConfigs (150.29s) |
04:51:35 crd_util.go:27: creating couchbase cluster: test-couchbase-6sjnz |
04:51:35 util.go:1357: context deadline exceeded: waiting for autoscaler `test-config-1.test-couchbase-6sjnz` to be created |
04:51:35 util.go:1358: goroutine 2840 [running]: |
04:51:35 runtime/debug.Stack(0x1ca2b8b, 0x0, 0xc00067bd50) |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/go/src/runtime/debug/stack.go:24 +0xab |
04:51:35 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc00035be00, 0x2613ae0, 0xc0000d4760) |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/test/e2e/e2eutil/util.go:1353 +0x34 |
04:51:35 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitUntilCouchbaseAutoscalerExists(0xc00035be00, 0xc000308600, 0xc000914500, 0xc000346600, 0x22, 0xdf8475800) |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/test/e2e/e2eutil/wait_util.go:1002 +0x9b |
04:51:35 github.com/couchbase/couchbase-operator/test/e2e.TestAutoscaleMultiConfigs(0xc00035be00) |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/test/e2e/autoscale_test.go:362 +0x3c2 |
04:51:35 testing.tRunner(0xc00035be00, 0x23aee70) |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/go/src/testing/testing.go:909 +0x19a |
04:51:35 created by testing.(*T).Run |
04:51:35 /jenkins/workspace/k8s-cbop-eks-pipeline/go/src/testing/testing.go:960 +0x652 |
The same generic error is repeated for all the impacted test cases.
Looking at the cbopinfo of failed test cases, the pods with couchbase-server.log had the following error:
Starting Couchbase Server -- Web UI available at http://<ip>:8091 |
and logs available in /opt/couchbase/var/lib/couchbase/logsThe maximum number of open files for the couchbase user is set too low. |
It must be at least 70000.Normally this can be increased by adding the following lines to |
/etc/security/limits.conf:couchbase soft nofile <value>
|
couchbase hard nofile <value>Where <value> is greater than 70000. The procedure may be totally |
different if you're running so called "non-root/non-sudo install" or |
if you've built Couchbase Server from source. |
Also, the job had the following warning all over the job console repeatedly continuously:
02:05:54 W1124 02:05:53.769074 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2331 clients constructed calling "aws" |
02:05:55 W1124 02:05:54.770617 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2332 clients constructed calling "aws" |
02:05:55 W1124 02:05:54.771280 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2333 clients constructed calling "aws" |
02:05:55 W1124 02:05:55.769074 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2334 clients constructed calling "aws" |
02:05:55 W1124 02:05:55.769659 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2335 clients constructed calling "aws" |
02:05:56 W1124 02:05:56.768485 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2336 clients constructed calling "aws" |
02:05:56 W1124 02:05:56.769182 17208 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 2337 clients constructed calling "aws" |
(cbopinfo of all failed test cases of said job attached, the logs for other jobs with desired versions can be fetched from the link provided above)
Note: The Autoscale tests did not fail on AKS/GCP platform with said versions.