[Linux] Operator does not bring up listed number of nodes.

This is highly likely an user error but if this is not a bug as per summary then the summary will be modified to report under-reporting in Operator logs.

Opensource K8s cluster v1.22.4. Status: 

[root@worker0 couchbase-autonomous-operator_2.3.0-292-kubernetes-linux-amd64]# kubectl get nodes

NAME      STATUS   ROLES                  AGE   VERSION

worker0   Ready    control-plane,master   46m   v1.22.4

worker1   Ready    <none>                 45m   v1.22.4

worker2   Ready    <none>                 45m   v1.22.4 

Steps to Reproduce:

• Bring up DAC and Operator Pod using Linux latest package.
• Create a Storage Class defined as: 

  apiVersion: storage.k8s.io/v1

  kind: StorageClass

  metadata:

  name: local-storage

  provisioner: kubernetes.io/no-provisioner

  volumeBindingMode: WaitForFirstConsumer

• Create a PV defined as: 

  apiVersion: v1

  kind: PersistentVolume

  metadata:

  name: pv

  spec:

  capacity:

  storage: 2Gi

  accessModes:

  - ReadWriteOnce

  persistentVolumeReclaimPolicy: Delete

  storageClassName: gp2

  local:

  path: /dev/xvdb

  nodeAffinity:

  required:

  nodeSelectorTerms:

  - matchExpressions:

  - key: kubernetes.io/hostname

  operator: In

  values:

  - worker2

  worker2/1 are my k8s nodes and /dev/xvdb is the disk device on all worker nodes.

• CB cluster is defined as: 

  spec:

  image: couchbase/server:7.0.3

  security:

  adminSecret: cb-example-auth

  buckets:

  managed: true

  servers:

  - size: 3

  name: all_services

  services:

  - data

  - index

  - query

  - search

  - eventing

  - analytics

  volumeMounts:

  default: couchbase

• Create the CB cluster.

Status of the pods: 

 [root@worker0 couchbase-autonomous-operator_2.3.0-292-kubernetes-linux-amd64]# kubectl get pods

NAME                                            READY   STATUS    RESTARTS   AGE

cb-example-0000                                 1/1     Running   0          7m54s

couchbase-operator-79664dfb6f-4vd2f             1/1     Running   0          19m

couchbase-operator-admission-74c76787d6-cbn5b   1/1     Running   0          112m

Status of PV: 

[root@worker0 couchbase-autonomous-operator_2.3.0-292-kubernetes-linux-amd64]# kubectl get pv

NAME   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE

pv     1Gi        RWO            Delete           Available           gp2                     18m 

* As you can see, even after waiting for ~8 minutes, we did not see all 3 CB pods come up.

• This is most likely because my PV definition is not sufficient and is probably erroneous which could explain only one of the pods coming up.
• If that's the case, why are we not seeing any suggestion of that in the logs?

Last few lines of operator logs: 

{"level":"info","ts":1648833356.4567714,"logger":"cluster","msg":"Cluster does not exist so the operator is attempting to create it","cluster":"default/cb-example"}

{"level":"info","ts":1648833361.4608867,"logger":"cluster","msg":"Couchbase client starting","cluster":"default/cb-example"}

{"level":"info","ts":1648833376.4845219,"logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"  (\n  \t\"\"\"\n  \t... // 13 identical lines\n  \t  status: \"False\"\n  \t  type: Available\n+ \tcurrentVersion: 7.0.3\n  \tmembers: {}\n  \tsize: 0\n  \t\"\"\"\n  )\n"}

{"level":"info","ts":1648833376.519635,"logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0000","image":"couchbase/server:7.0.3"} 

Full Operator logs attached.