Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
2.2.3
-
1
Description
Deploying a CB cluster on K8s with CAO 2.2.3
(using Helm chart) I have lots of unstability.
The CAO keeps crashing (multiple CrashLoopBackOff in series) with this error message:
~ » kubectl logs -f eks-helm-couchbase-operator-57856d798-p2trg -n dev-couchbase
{"level":"info","ts":1655251918.987866,"logger":"main","msg":"couchbase-operator","version":"2.2.3 (build 102)","revision":"e344b99063ebfcd1b2bd07297935ff5fbc03211b"} {"level":"info","ts":1655251919.5946047,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1655251919.5956771,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1655251919.595681,"msg":"attempting to acquire leader lease dev-couchbase/couchbase-operator...\n"} {"level":"info","ts":1655251937.0544872,"msg":"successfully acquired lease dev-couchbase/couchbase-operator\n"} {"level":"info","ts":1655251937.054694,"logger":"controller","msg":"Starting EventSource","controller":"couchbase-controller","source":"kind source: /, Kind="} {"level":"info","ts":1655251937.1557167,"logger":"controller","msg":"Starting Controller","controller":"couchbase-controller"} {"level":"info","ts":1655251937.15576,"logger":"controller","msg":"Starting workers","controller":"couchbase-controller","worker count":4} {"level":"info","ts":1655251937.1560667,"logger":"cluster","msg":"Watching new cluster","cluster":"dev-couchbase/eks-helm-couchbase-cluster"} {"level":"info","ts":1655251954.194371,"logger":"cluster","msg":"Couchbase client starting","cluster":"dev-couchbase/eks-helm-couchbase-cluster"} {"level":"info","ts":1655251954.1962802,"logger":"cluster","msg":"Running","cluster":"dev-couchbase/eks-helm-couchbase-cluster"} {"level":"info","ts":1655251954.196843,"logger":"cluster","msg":"Janitor starting","cluster":"dev-couchbase/eks-helm-couchbase-cluster"} {"level":"info","ts":1655251954.3479068,"logger":"cluster","msg":"Cluster status","cluster":"dev-couchbase/eks-helm-couchbase-cluster","balance":"unbalanced","rebalancing":false} {"level":"info","ts":1655251954.3479586,"logger":"cluster","msg":"Node status","cluster":"dev-couchbase/eks-helm-couchbase-cluster","name":"eks-helm-couchbase-cluster-0000","version":"7.0.2","class":"default","managed":true,"status":"Active"} {"level":"info","ts":1655251954.3479726,"logger":"cluster","msg":"Node status","cluster":"dev-couchbase/eks-helm-couchbase-cluster","name":"eks-helm-couchbase-cluster-0001","version":"7.0.2","class":"default","managed":true,"status":"PendingAdd"} {"level":"info","ts":1655251954.347977,"logger":"cluster","msg":"Node status","cluster":"dev-couchbase/eks-helm-couchbase-cluster","name":"eks-helm-couchbase-cluster-0002","version":"7.0.2","class":"default","managed":true,"status":"PendingAdd"} {"level":"error","ts":1655251956.0474403,"msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 249 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic(0x15d9ee0, 0x22de220)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/runtime/runtime.go:74 +0xa6\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/runtime/runtime.go:48 +0x86\npanic(0x15d9ee0, 0x22de220)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangqOHki/go1.16.3/src/runtime/panic.go:965 +0x1b9\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMemberAlternateAddresses(0xc000436480, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:1205 +0x6b\ngithub.com/couchbase/couchbase-operator/pkg/cluster.handleNodeServices(0xc0003fafc0, 0xc000436480, 0x183d5a0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:1089 +0x50\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step(0xc0003fafc0, 0xc000436480, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:264 +0x6e\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec(0xc0003fafc0, 0xc000436480, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:280 +0x47\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers(...)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:276\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile(0xc000436480, 0x0, 0x0)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:165 +0x5d3\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile(0xc000436480)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:511 +0xdaf\ngithub.com/couchbase/couchbase-operator/pkg/cluster.New(0x7fffe585267f, 0x3, 0xc00044ab00, 0xc00028a0c0, 0x2, 0x2)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:205 +0x7ff\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile(0xc0003383f0, 0xc000531780, 0xd, 0xc000512ac0, 0x1a, 0xc00083d920, 0xc000242240, 0xc000172128, 0xc000172120)\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/controller/controller.go:74 +0xbb5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000468000, 0x1630300, 0xc000250000, 0x810c00)\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:244 +0x2a9\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000468000, 0x203000)\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:218 +0xb0\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00083d8f0)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:155 +0x5f\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00083d8f0, 0x19595a0, 0xc0000c61e0, 0x1, 0xc00046b860)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:156 +0x9b\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00083d8f0, 0x3b9aca00, 0x0, 0xc000502101, 0xc00046b860)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:133 +0x98\nk8s.io/apimachinery/pkg/util/wait.Until(0xc00083d8f0, 0x3b9aca00, 0xc00046b860)\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:90 +0x4d\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:179 +0x3d6\n","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132\nk8s.io/klog/v2.(*loggingT).output\n\t/home/couchbase/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:893\nk8s.io/klog/v2.(*loggingT).printf\n\t/home/couchbase/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:733\nk8s.io/klog/v2.Errorf\n\t/home/couchbase/go/pkg/mod/k8s.io/klog/v2@v2.2.0/klog.go:1416\nk8s.io/apimachinery/pkg/util/runtime.logPanic\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/runtime/runtime.go:78\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/runtime/runtime.go:48\nruntime.gopanic\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangqOHki/go1.16.3/src/runtime/panic.go:965\nruntime.panicmem\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangqOHki/go1.16.3/src/runtime/panic.go:212\nruntime.sigpanic\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangqOHki/go1.16.3/src/runtime/signal_unix.go:734\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMemberAlternateAddresses\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:1205\ngithub.com/couchbase/couchbase-operator/pkg/cluster.handleNodeServices\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:1089\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:264\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:280\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:276\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:165\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:511\ngithub.com/couchbase/couchbase-operator/pkg/cluster.New\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:205\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/controller/controller.go:74\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:90"}panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x143ca4b]goroutine 249 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/runtime/runtime.go:55 +0x109
panic(0x15d9ee0, 0x22de220)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangqOHki/go1.16.3/src/runtime/panic.go:965 +0x1b9
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMemberAlternateAddresses(0xc000436480, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:1205 +0x6b
github.com/couchbase/couchbase-operator/pkg/cluster.handleNodeServices(0xc0003fafc0, 0xc000436480, 0x183d5a0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:1089 +0x50
github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).step(0xc0003fafc0, 0xc000436480, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:264 +0x6e
github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec(0xc0003fafc0, 0xc000436480, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/nodereconcile.go:280 +0x47
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers(...)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:276
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile(0xc000436480, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:165 +0x5d3
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile(0xc000436480)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:511 +0xdaf
github.com/couchbase/couchbase-operator/pkg/cluster.New(0x7fffe585267f, 0x3, 0xc00044ab00, 0xc00028a0c0, 0x2, 0x2)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:205 +0x7ff
github.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile(0xc0003383f0, 0xc000531780, 0xd, 0xc000512ac0, 0x1a, 0xc00083d920, 0xc000242240, 0xc000172128, 0xc000172120)
/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/controller/controller.go:74 +0xbb5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000468000, 0x1630300, 0xc000250000, 0x810c00)
/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:244 +0x2a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000468000, 0x203000)
/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:218 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00083d8f0)
/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00083d8f0, 0x19595a0, 0xc0000c61e0, 0x1, 0xc00046b860)
/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00083d8f0, 0x3b9aca00, 0x0, 0xc000502101, 0xc00046b860)
/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00083d8f0, 0x3b9aca00, 0xc00046b860)
/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/internal/controller/controller.go:179 +0x3d6
As a consequence the cluster is never rebalanced (hence the alternate addresses are not created (I notice they are only created after the rebalance is successful and this time it never happens).
Current case is using Public Networking with external DNS.
(AWS / Cloudflare as DDNS and Namecheap as DNS provider)