[CX] Rebalance reports success even though it failed

2020-10-07T14:59:34.803-07:00 INFO UpgradeTestBase [main] Recreating 'cc' node (image = [build-docker.couchbase.com/couchbase/server-internal:6.6.1-9104], forceNewIp = false)... 2020-10-07T14:59:34.807-07:00 INFO DockerTestBase [main] Stopping docker container: fa468d77d00a94a66511700c4f7266e9ca84cb987bfe71149531511f3296ee01 2020-10-07T14:59:37.769-07:00 INFO DockerTestBase [main] Stopped docker container: fa468d77d00a94a66511700c4f7266e9ca84cb987bfe71149531511f3296ee01 2020-10-07T14:59:37.769-07:00 INFO DockerTestBase [main] Removing docker container: fa468d77d00a94a66511700c4f7266e9ca84cb987bfe71149531511f3296ee01 2020-10-07T14:59:37.847-07:00 INFO DockerTestBase [main] Removed docker container: fa468d77d00a94a66511700c4f7266e9ca84cb987bfe71149531511f3296ee01 2020-10-07T14:59:37.858-07:00 INFO DockerTestBase [main] Creating docker container (test dir = Couchbase/mad_hatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0005_cc_build-docker.couchbase.com_couchbase_server-internal_6.6.1-9104, network = { name = cbas-docker-test-0001-2e43f5f6-162e-4c6e-98a6-66f298732ada, id = 43d9db2cdcc2c89d07dd3ecc64164010673062af4b93ed4bf3307dca7dd03b94 }) 2020-10-07T14:59:39.705-07:00 INFO DockerTestBase [main] Container information for d34c7c8298f47f1762575b7711fcc0b31f2a717f50f8cbd284f80a179d6e0f0d: host: cc.couchbase.host, ip: 192.168.176.3, 127.0.0.1:34477->192.168.176.3:8091, 127.0.0.1:34473->192.168.176.3:8095 2020-10-07T14:59:59.110-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {REBALANCE_REQUIRED} for up to 360s... 2020-10-07T15:00:56.210-07:00 INFO UpgradeTestBase [main] Recreating 'nc' node (image = [build-docker.couchbase.com/couchbase/server-internal:6.6.1-9104], forceNewIp = false)... 2020-10-07T15:00:56.210-07:00 INFO DockerTestBase [main] Stopping docker container: c26c376ad6c897b3a659d0fb8d6f443b3ceac4347d754019d44c3836d6322ce5 2020-10-07T15:00:58.148-07:00 INFO DockerTestBase [main] Stopped docker container: c26c376ad6c897b3a659d0fb8d6f443b3ceac4347d754019d44c3836d6322ce5 2020-10-07T15:00:58.148-07:00 INFO DockerTestBase [main] Removing docker container: c26c376ad6c897b3a659d0fb8d6f443b3ceac4347d754019d44c3836d6322ce5 2020-10-07T15:00:58.210-07:00 INFO DockerTestBase [main] Removed docker container: c26c376ad6c897b3a659d0fb8d6f443b3ceac4347d754019d44c3836d6322ce5 2020-10-07T15:00:58.211-07:00 INFO DockerTestBase [main] Creating docker container (test dir = Couchbase/mad_hatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0006_nc_build-docker.couchbase.com_couchbase_server-internal_6.6.1-9104, network = { name = cbas-docker-test-0001-2e43f5f6-162e-4c6e-98a6-66f298732ada, id = 43d9db2cdcc2c89d07dd3ecc64164010673062af4b93ed4bf3307dca7dd03b94 }) 2020-10-07T15:01:00.573-07:00 INFO DockerTestBase [main] Container information for 2297e4111fef57a230409d2be28d8f98ad4c4cffb5b76a08f5475686422a03c5: host: nc.couchbase.host, ip: 192.168.176.4, 127.0.0.1:34492->192.168.176.4:8091, 127.0.0.1:34488->192.168.176.4:8095 2020-10-07T15:01:26.858-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {REBALANCE_REQUIRED} for up to 360s... 2020-10-07T15:01:26.978-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 6.6.x' condition for up to 60 seconds... 2020-10-07T15:01:30.309-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2020-10-07T15:01:30.309-07:00 INFO DockerTestBase [main] Executing the command: docker exec d091656842ead482a83b56b484da13309007b20a79c41a8f05b4150815b4a5f8 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.5:8091 -u couchbase -p couchbase 2020-10-07T15:07:37.022-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2020-10-07T15:07:37.022-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2020-10-07T15:07:37.547-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 60 seconds... 2020-10-07T15:07:37.617-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2020-10-07T15:07:37.618-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 360s... 2020-10-07T15:13:37.652-07:00 WARN DockerTestBase [main] Timed out waiting for condition: NCs == 2 (last count was 2) & Cluster [ACTIVE] (last state was REBALANCE_REQUIRED), attemptCount: 330