Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-655

RZA: Nodes not schedulled in the round robin manner with respect to server-group order

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.1.0
    • testing

    Description

      Testcase: TestRzaCreateClusterWithClassBasedConfig

      Reproducible: Not every time

      operator-image: couchbase/couchbase-operator-internal:1.1.0-518

      Cluster server config spec:

       

        servers:
        - name: test_config_2
          pod:
            resources: {}
          serverGroups:
          - RzaGroup-2
          services:
          - query
          size: 1
        - name: test_config_3
          pod:
            resources: {}
          serverGroups:
          - RzaGroup-1
          - RzaGroup-3
          services:
          - search
          size: 3
        - name: test_config_1
          pod:
            resources: {}
          serverGroups:
          - RzaGroup-1
          - RzaGroup-3
          services:
          - data
          - index
          size: 3

      K8S nodes and pod status:

       

       

      couchbase-operator]$ kubectl get pods -o wide
      NAME                                 READY     STATUS    RESTARTS   AGE       IP          NODE
      couchbase-operator-777b86496-bgrdh   1/1       Running   0          19m       10.32.0.4   k8s-worker-2
      test-couchbase-flvcl-0000            1/1       Running   0          19m       10.38.0.2   k8s-worker-1
      test-couchbase-flvcl-0001            1/1       Running   0          18m       10.32.0.2   k8s-worker-2
      test-couchbase-flvcl-0002            1/1       Running   0          17m       10.38.0.3   k8s-worker-1
      test-couchbase-flvcl-0003            1/1       Running   0          17m       10.44.0.3   k8s-worker-3
      test-couchbase-flvcl-0004            1/1       Running   0          16m       10.38.0.4   k8s-worker-1
      test-couchbase-flvcl-0005            1/1       Running   0          15m       10.44.0.4   k8s-worker-3
      test-couchbase-flvcl-0006            1/1       Running   0          14m       10.38.0.5   k8s-worker-1
       
      couchbase-operator]$ kubectl get nodes --show-labels
      NAME           STATUS    ROLES     AGE       VERSION   LABELS
      k8s-master-1   Ready     master    2h        v1.10.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=RzaGroup-2,kubernetes.io/hostname=k8s-master-1,node-role.kubernetes.io/master=,px/enabled=true
      k8s-worker-1   Ready     <none>    2h        v1.10.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=RzaGroup-1,kubernetes.io/hostname=k8s-worker-1,px/enabled=true
      k8s-worker-2   Ready     <none>    2h        v1.10.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=RzaGroup-2,kubernetes.io/hostname=k8s-worker-2,px/enabled=true
      k8s-worker-3   Ready     <none>    2h        v1.10.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=RzaGroup-3,kubernetes.io/hostname=k8s-worker-3,px/enabled=true

       

      Expected and actual deployment map:

       

      Deployed mapping:
       test-couchbase-flvcl-0000:RzaGroup-1
       test-couchbase-flvcl-0001:RzaGroup-1
       test-couchbase-flvcl-0002:RzaGroup-3
       test-couchbase-flvcl-0003:RzaGroup-1
       test-couchbase-flvcl-0004:RzaGroup-3
       test-couchbase-flvcl-0005:RzaGroup-1
       test-couchbase-flvcl-0006:RzaGroup-2
       
      Expected deployment mapping:
       test-couchbase-flvcl-0000:RzaGroup-1
       test-couchbase-flvcl-0001:RzaGroup-2 <- Expected on RzaGroup3 (test_config_1)
       test-couchbase-flvcl-0002:RzaGroup-1
       test-couchbase-flvcl-0003:RzaGroup-2
       test-couchbase-flvcl-0004:RzaGroup-1
       test-couchbase-flvcl-0005:RzaGroup-3
       test-couchbase-flvcl-0006:RzaGroup-1
      
      

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment -

          ashwin/pod/test-couchbase-flvcl-0000/test-couchbase-flvcl-0000.yaml:    couchbase_node_conf: test_config_1
          ashwin/pod/test-couchbase-flvcl-0000/test-couchbase-flvcl-0000.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-1
           
          ashwin/pod/test-couchbase-flvcl-0001/test-couchbase-flvcl-0001.yaml:    couchbase_node_conf: test_config_2
          ashwin/pod/test-couchbase-flvcl-0001/test-couchbase-flvcl-0001.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-2
           
          ashwin/pod/test-couchbase-flvcl-0002/test-couchbase-flvcl-0002.yaml:    couchbase_node_conf: test_config_3
          ashwin/pod/test-couchbase-flvcl-0002/test-couchbase-flvcl-0002.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-1
          ashwin/pod/test-couchbase-flvcl-0003/test-couchbase-flvcl-0003.yaml:    couchbase_node_conf: test_config_3
          ashwin/pod/test-couchbase-flvcl-0003/test-couchbase-flvcl-0003.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-3
          ashwin/pod/test-couchbase-flvcl-0004/test-couchbase-flvcl-0004.yaml:    couchbase_node_conf: test_config_3
          ashwin/pod/test-couchbase-flvcl-0004/test-couchbase-flvcl-0004.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-1
           
          ashwin/pod/test-couchbase-flvcl-0005/test-couchbase-flvcl-0005.yaml:    couchbase_node_conf: test_config_1
          ashwin/pod/test-couchbase-flvcl-0005/test-couchbase-flvcl-0005.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-3
          ashwin/pod/test-couchbase-flvcl-0006/test-couchbase-flvcl-0006.yaml:    couchbase_node_conf: test_config_1
          ashwin/pod/test-couchbase-flvcl-0006/test-couchbase-flvcl-0006.yaml:    failure-domain.beta.kubernetes.io/zone: RzaGroup-1

          Pod 0 comes up first from test_config_1 as it has the data service (zones 1,3)

          Then pod 1 comes up from test_config_2 as it's the first in the list of server configs (zone 2)

          Then pods 2-4 come up from test_config_3 as it's next in the list (zones 1,3)

          Finally pods 5-6 come up from test_config_1 as it's last in the list (zones 1,3)

          As far as I can see the system is working as designed, server groups are correctly balanced and the test is wrong.

           

          simon.murray Simon Murray added a comment - ashwin/pod/test-couchbase-flvcl-0000/test-couchbase-flvcl-0000.yaml: couchbase_node_conf: test_config_1 ashwin/pod/test-couchbase-flvcl-0000/test-couchbase-flvcl-0000.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-1   ashwin/pod/test-couchbase-flvcl-0001/test-couchbase-flvcl-0001.yaml: couchbase_node_conf: test_config_2 ashwin/pod/test-couchbase-flvcl-0001/test-couchbase-flvcl-0001.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-2   ashwin/pod/test-couchbase-flvcl-0002/test-couchbase-flvcl-0002.yaml: couchbase_node_conf: test_config_3 ashwin/pod/test-couchbase-flvcl-0002/test-couchbase-flvcl-0002.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-1 ashwin/pod/test-couchbase-flvcl-0003/test-couchbase-flvcl-0003.yaml: couchbase_node_conf: test_config_3 ashwin/pod/test-couchbase-flvcl-0003/test-couchbase-flvcl-0003.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-3 ashwin/pod/test-couchbase-flvcl-0004/test-couchbase-flvcl-0004.yaml: couchbase_node_conf: test_config_3 ashwin/pod/test-couchbase-flvcl-0004/test-couchbase-flvcl-0004.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-1   ashwin/pod/test-couchbase-flvcl-0005/test-couchbase-flvcl-0005.yaml: couchbase_node_conf: test_config_1 ashwin/pod/test-couchbase-flvcl-0005/test-couchbase-flvcl-0005.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-3 ashwin/pod/test-couchbase-flvcl-0006/test-couchbase-flvcl-0006.yaml: couchbase_node_conf: test_config_1 ashwin/pod/test-couchbase-flvcl-0006/test-couchbase-flvcl-0006.yaml: failure-domain.beta.kubernetes.io/zone: RzaGroup-1 Pod 0 comes up first from test_config_1 as it has the data service (zones 1,3) Then pod 1 comes up from test_config_2 as it's the first in the list of server configs (zone 2) Then pods 2-4 come up from test_config_3 as it's next in the list (zones 1,3) Finally pods 5-6 come up from test_config_1 as it's last in the list (zones 1,3) As far as I can see the system is working as designed, server groups are correctly balanced and the test is wrong.  

          Added Sort function in crd.go to overcome this map issue.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Added Sort function in crd.go to overcome this map issue.

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty