Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-357

WaitClusterStatusHealthy api call fails with wrong results

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.0.0
    • None
    • operator

    Description

      TestAntiAffinityOnCannotBeScaled case fails due to  WaitClusterStatusHealthy returns cluster status as unhealthy even without any cluster changes after ClusterCreate call.

      --- FAIL: TestAntiAffinityOnCannotBeScaled (460.17s)
              pod_test.go:328: AntiAffinity=on...
                       attempting to create 1 pod cluster with 1 nodes
              crd_util.go:42: creating couchbase cluster: test-couchbase-nt94f
              util.go:520: 2018-06-04 14:23:16.946017321 +0530 IST m=+31.662409802 waiting size (1), healthy couchbase members: names ([])
              util.go:520: 2018-06-04 14:23:26.948412652 +0530 IST m=+41.664698957 waiting size (1), healthy couchbase members: names ([])
              util.go:520: 2018-06-04 14:23:36.948146495 +0530 IST m=+51.664349627 waiting size (1), healthy couchbase members: names ([])
              util.go:520: 2018-06-04 14:23:46.948899627 +0530 IST m=+61.665031626 waiting size (1), healthy couchbase members: names ([])
              util.go:520: 2018-06-04 14:23:56.950883629 +0530 IST m=+71.666950806 waiting size (1), healthy couchbase members: names ([])
              util.go:520: 2018-06-04 14:24:06.949918139 +0530 IST m=+81.665923825 waiting size (1), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              pod_test.go:335: Cluster created
              pod_test.go:349: Attempting to add a node
              util.go:544: Changing Cluster Size To: 2...
              util.go:553: Waiting For Cluster Size To Be: 2...
              util.go:520: 2018-06-04 14:24:06.964623272 +0530 IST m=+81.680628869 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:24:16.968369523 +0530 IST m=+91.684315343 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:24:26.966567703 +0530 IST m=+101.682454688 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:24:36.968342511 +0530 IST m=+111.684171117 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:24:46.965597126 +0530 IST m=+121.681367631 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:24:56.96850726 +0530 IST m=+131.684219765 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:06.968787434 +0530 IST m=+141.684442022 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:16.970448399 +0530 IST m=+151.686045098 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:26.967432251 +0530 IST m=+161.682971104 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:36.969310102 +0530 IST m=+171.684791093 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:46.970007799 +0530 IST m=+181.685430940 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:25:56.969531086 +0530 IST m=+191.684896388 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:06.96940827 +0530 IST m=+201.684715733 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:16.970063801 +0530 IST m=+211.685313421 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:26.966540686 +0530 IST m=+221.681732483 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:36.96972426 +0530 IST m=+231.684858195 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:46.969779427 +0530 IST m=+241.684855518 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:26:56.970814269 +0530 IST m=+251.685832510 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:06.968945029 +0530 IST m=+261.683905438 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:16.968718313 +0530 IST m=+271.683620880 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:26.968095978 +0530 IST m=+281.682940705 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:36.969459248 +0530 IST m=+291.684246123 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:46.970913757 +0530 IST m=+301.685642780 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:27:56.968596282 +0530 IST m=+311.683267475 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:06.96838037 +0530 IST m=+321.682993721 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:16.967899544 +0530 IST m=+331.682455054 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:26.969948826 +0530 IST m=+341.684446481 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:36.968857935 +0530 IST m=+351.683297753 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:46.969468861 +0530 IST m=+361.683850832 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:28:56.968585765 +0530 IST m=+371.682909898 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:520: 2018-06-04 14:29:06.966782667 +0530 IST m=+381.681048967 waiting size (2), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              pod_test.go:354: Node not added
              pod_test.go:358: Reverting add
              util.go:544: Changing Cluster Size To: 1...
              util.go:553: Waiting For Cluster Size To Be: 1...
              util.go:520: 2018-06-04 14:29:06.980448631 +0530 IST m=+381.694714851 waiting size (1), healthy couchbase members: names ([test-couchbase-nt94f-0000])
              util.go:558: Resize Success: [test-couchbase-nt94f-0000]...
              util.go:520: 2018-06-04 14:29:06.984576152 +0530 IST m=+381.698842349 Cluster Status Conditions: (map[Scaling:0xc42060c910 Available:0xc42060c870 Balanced:0xc42060c8c0])
              util.go:520: 2018-06-04 14:29:06.984677781 +0530 IST m=+381.698943977 Cluster scaling ...
              util.go:520: 2018-06-04 14:29:16.98975056 +0530 IST m=+391.703958883 Cluster Status Conditions: (map[Balanced:0xc42060ca00 Scaling:0xc42060ca50 Available:0xc42060c9b0])
              util.go:520: 2018-06-04 14:29:16.98986066 +0530 IST m=+391.704068982 Cluster scaling ...
              util.go:520: 2018-06-04 14:29:26.988795135 +0530 IST m=+401.702945620 Cluster Status Conditions: (map[Available:0xc4200dae60 Balanced:0xc4200daeb0 Scaling:0xc4200daf00])
              util.go:520: 2018-06-04 14:29:26.988949596 +0530 IST m=+401.703100080 Cluster scaling ...
              util.go:520: 2018-06-04 14:29:36.988685238 +0530 IST m=+411.702777880 Cluster Status Conditions: (map[Balanced:0xc42060cb40 Scaling:0xc42060cb90 Available:0xc42060caf0])
              util.go:520: 2018-06-04 14:29:36.988762211 +0530 IST m=+411.702854853 Cluster scaling ...
              util.go:520: 2018-06-04 14:29:46.98894889 +0530 IST m=+421.702983687 Cluster Status Conditions: (map[Available:0xc420250c80 Balanced:0xc420250cd0 Scaling:0xc420250d20])
              util.go:520: 2018-06-04 14:29:46.989039888 +0530 IST m=+421.703074685 Cluster scaling ...
              util.go:520: 2018-06-04 14:29:56.989431601 +0530 IST m=+431.703408552 Cluster Status Conditions: (map[Available:0xc42060c050 Balanced:0xc42060c0a0 Scaling:0xc42060c190])
              util.go:520: 2018-06-04 14:29:56.989506227 +0530 IST m=+431.703483177 Cluster scaling ...
              util.go:520: 2018-06-04 14:30:06.990030426 +0530 IST m=+441.703949530 Cluster Status Conditions: (map[Scaling:0xc42044a230 Available:0xc42044a190 Balanced:0xc42044a1e0])
              util.go:520: 2018-06-04 14:30:06.990114993 +0530 IST m=+441.704034096 Cluster scaling ...
              util.go:520: 2018-06-04 14:30:16.988135216 +0530 IST m=+451.701996487 Cluster Status Conditions: (map[Available:0xc42044a370 Balanced:0xc42044a3c0 Scaling:0xc42044a460])
              util.go:520: 2018-06-04 14:30:16.988206027 +0530 IST m=+451.702067298 Cluster scaling ...
              util.go:520: 2018-06-04 14:30:26.988349007 +0530 IST m=+461.702152433 Cluster Status Conditions: (map[Available:0xc42044a500 Balanced:0xc42044a550 Scaling:0xc42044a5a0])
              util.go:520: 2018-06-04 14:30:26.98841207 +0530 IST m=+461.702215496 Cluster scaling ...
              util.go:520: 2018-06-04 14:30:36.98816481 +0530 IST m=+471.701910394 Cluster Status Conditions: (map[Available:0xc42044a640 Balanced:0xc42044a690 Scaling:0xc42044a6e0])
              util.go:520: 2018-06-04 14:30:36.988263159 +0530 IST m=+471.702008742 Cluster scaling ...
              util.go:520: 2018-06-04 14:30:46.988521015 +0530 IST m=+481.702208753 Cluster Status Conditions: (map[Available:0xc420250550 Balanced:0xc4202505a0 Scaling:0xc420250640])
              util.go:520: 2018-06-04 14:30:46.988605477 +0530 IST m=+481.702293215 Cluster scaling ...
              pod_test.go:367: fail to wait for cluster status to be healthy: still failing after 10 retries
       
              util.go:452: Deleting clusters: [[test-couchbase-nt94f]]
              util.go:454: Attempting to delete: [test-couchbase-nt94f]
              util.go:459: Successfully deleted: [test-couchbase-nt94f]
              util.go:466: Killing pods: [test-couchbase-nt94f-0000]
              util.go:470: Waiting for deletion of pod: test-couchbase-nt94f-0000

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              simon.murray Simon Murray
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty