![[Jenkins]](/static/a526f9ea/images/headshot.png)
Regression
com.couchbase.analytics.test.cbserver.ClusterRebalanceIT.test[ClusterRebalanceIT 10: rebalance: rebalance_MB-28183]
Failing for the past 1 build
(Since 
#7980 )
#7980 )
add descriptionError Message
couchbase-cli non-zero exit code; output: ERROR: Internal server error, please retry your request expected:<0> but was:<1>
Stacktrace
java.lang.AssertionError: couchbase-cli non-zero exit code; output: ERROR: Internal server error, please retry your request expected:<0> but was:<1> at com.couchbase.analytics.test.cbserver.ClusterRebalanceIT.test(ClusterRebalanceIT.java:29)
Standard Error
2019-06-14T22:36:40.875-07:00 WARN TestMethodTracer [main] ### test[ClusterRebalanceIT 10: rebalance: rebalance_MB-28183] START
2019-06-14T22:36:40.988-07:00 INFO TestExecutor [main] Starting [TEST]: rebalance/rebalance_MB-28183 ...
2019-06-14T22:36:40.989-07:00 INFO ClusterExecutionITBase [main] starting cluster_run
2019-06-14T22:36:48.021-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0)
2019-06-14T22:36:48.309-07:00 INFO ClusterExecutionITBase [main] cluster is running
2019-06-14T22:36:48.310-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA]
2019-06-14T22:36:48.310-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir]
2019-06-14T22:36:50.780-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized
2019-06-14T22:36:50.797-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072]
2019-06-14T22:36:51.048-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized
2019-06-14T22:36:51.068-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA},{},{},{}}, cbas node count: 0
2019-06-14T22:36:51.074-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip
2019-06-14T22:36:58.343-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details.
2019-06-14T22:36:58.346-07:00 INFO TestExecutor [main] cbdocloader exit code 1
2019-06-14T22:36:58.347-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0]
2019-06-14T22:36:58.360-07:00 INFO ClusterExecutionITBase [main] Nothing to do, requested topology already in place: [0]
2019-06-14T22:36:58.360-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1]
2019-06-14T22:36:58.372-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1]]) are fully removed
2019-06-14T22:36:58.379-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: [])
2019-06-14T22:36:58.380-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1] with service(s): [ANALYTICS]
2019-06-14T22:37:01.066-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized
2019-06-14T22:37:01.103-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, http://127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics]
2019-06-14T22:37:04.754-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added
2019-06-14T22:37:04.776-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase]
2019-06-14T22:37:06.982-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os
2019-06-14T22:37:06.983-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete
2019-06-14T22:37:07.000-07:00 INFO ClusterExecutionITBase [main] state: [0, 1], nodes: {{DATA},{ANALYTICS},{},{}}, cbas node count: 1
2019-06-14T22:37:07.000-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-42-/127.0.0.1:9615] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 60s...
2019-06-14T22:37:15.883-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1] to [0, 1, 2, 3]
2019-06-14T22:37:15.890-07:00 INFO ClusterExecutionITBase [main] Updating cluster address for n_1 to 172.18.0.2:9001
2019-06-14T22:37:15.891-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[2, 3]]) are fully removed
2019-06-14T22:37:15.896-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: [])
2019-06-14T22:37:15.896-07:00 INFO ClusterExecutionITBase [main] adding node(s) [2, 3] with service(s): [ANALYTICS]
2019-06-14T22:37:18.621-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized
2019-06-14T22:37:18.739-07:00 INFO ClusterExecutionITBase [main -> init n_3+] >> SUCCESS: Node initialized
2019-06-14T22:37:18.753-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, http://127.0.0.1:9002,http://127.0.0.1:9003, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics]
2019-06-14T22:37:23.990-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added
2019-06-14T22:37:24.000-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase]
2019-06-14T22:37:33.270-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os
2019-06-14T22:37:33.271-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete
2019-06-14T22:37:33.286-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2, 3], nodes: {{DATA},{ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3
2019-06-14T22:37:33.288-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-43-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster state in {ACTIVE} for up to 60s...
2019-06-14T22:37:33.288-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-43-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster state in {ACTIVE} for up to 60s...
2019-06-14T22:37:33.288-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-43-/127.0.0.1:9645] Waiting for NCs == 3 & Cluster state in {ACTIVE} for up to 60s...
2019-06-14T22:37:40.273-07:00 INFO TestExecutor [pool-225-thread-1] Expected results file: src/test/resources/runtimets/results/rebalance/rebalance_MB-28183/rebalance_MB-28183.5.adm
2019-06-14T22:37:40.273-07:00 INFO TestExecutor [pool-225-thread-1] Actual results file: target/rttest/results/rebalance/rebalance_MB-28183/rebalance_MB-28183.5.adm encoding: UTF-8
2019-06-14T22:37:40.273-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rebalance/rebalance_MB-28183/rebalance_MB-28183.4.pollquery.sqlpp changed at line 1:
expected < { "count": 5891 }
actual > { "count": 1502 }
2019-06-14T22:37:41.274-07:00 INFO TestExecutor [pool-225-thread-1] using Accept-Charset: UTF-32LE
2019-06-14T22:37:42.049-07:00 INFO TestExecutor [pool-225-thread-1] Expected results file: src/test/resources/runtimets/results/rebalance/rebalance_MB-28183/rebalance_MB-28183.5.adm
2019-06-14T22:37:42.049-07:00 INFO TestExecutor [pool-225-thread-1] Actual results file: target/rttest/results/rebalance/rebalance_MB-28183/rebalance_MB-28183.5.adm encoding: UTF-32LE
2019-06-14T22:37:42.050-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rebalance/rebalance_MB-28183/rebalance_MB-28183.4.pollquery.sqlpp changed at line 1:
expected < { "count": 5891 }
actual > { "count": 3422 }
2019-06-14T22:37:43.051-07:00 INFO TestExecutor [pool-225-thread-1] using Accept-Charset: UTF-16
2019-06-14T22:37:43.486-07:00 INFO TestExecutor [pool-225-thread-1] Expected results file: src/test/resources/runtimets/results/rebalance/rebalance_MB-28183/rebalance_MB-28183.5.adm
2019-06-14T22:37:43.487-07:00 INFO TestExecutor [main] Starting loop '2 times back to 'rebalance_MB-28183.1.cb.cmd'...
2019-06-14T22:37:43.488-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1, 2, 3] to [0]
2019-06-14T22:37:43.500-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([]) are fully removed
2019-06-14T22:37:43.501-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: [])
2019-06-14T22:37:43.501-07:00 INFO ClusterExecutionITBase [main] removing nodes [1, 2, 3]
2019-06-14T22:37:43.502-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-remove, 172.18.0.2:9001,127.0.0.1:9002,127.0.0.1:9003]
2019-06-14T22:37:46.702-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os
2019-06-14T22:37:46.702-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete
2019-06-14T22:37:46.757-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA},{},{},{}}, cbas node count: 0
2019-06-14T22:37:46.757-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1]
2019-06-14T22:37:46.772-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1]]) are fully removed
2019-06-14T22:37:49.783-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: [])
2019-06-14T22:37:49.783-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1] with service(s): [ANALYTICS]
2019-06-14T22:37:49.943-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized
2019-06-14T22:37:49.965-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, http://172.18.0.2:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics]
2019-06-14T22:37:59.972-07:00 INFO ClusterExecutionITBase [main+] >> ERROR: Internal server error, please retry your request
2019-06-14T22:37:59.994-07:00 INFO TestExecutor [main] Cleaning up after test case rebalance: rebalance_MB-28183...
2019-06-14T22:37:59.994-07:00 INFO TestExecutor [main] Skipping clean up for test case rebalance: rebalance_MB-28183 since no analytics nodes are active
2019-06-14T22:37:59.994-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run...
2019-06-14T22:38:00.022-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT
2019-06-14T22:38:00.105-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited
2019-06-14T22:38:00.111-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds...
2019-06-14T22:38:00.111-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring cluster ports are not active
2019-06-14T22:38:02.122-07:00 INFO ClusterRebalanceIT [main] Copying logs+data to 'target/ClusterRebalanceIT/test[ClusterRebalanceIT 10: rebalance: rebalance_MB-28183]'...
2019-06-14T22:38:02.267-07:00 WARN TestMethodTracer [main] ### test[ClusterRebalanceIT 10: rebalance: rebalance_MB-28183] FAILED
java.lang.AssertionError: couchbase-cli non-zero exit code; output:
ERROR: Internal server error, please retry your request expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88) ~[junit-4.12.jar:4.12]
at org.junit.Assert.failNotEquals(Assert.java:834) ~[junit-4.12.jar:4.12]
at org.junit.Assert.assertEquals(Assert.java:645) ~[junit-4.12.jar:4.12]
at com.couchbase.analytics.test.common.ClusterTestBase.runCli(ClusterTestBase.java:714) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.common.ClusterTestBase.addNodes(ClusterTestBase.java:1132) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.common.ClusterTestBase$ClusterHelperImpl.rebalanceTo(ClusterTestBase.java:1326) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.CBASTestExecutor.executeCouchbaseCommand(CBASTestExecutor.java:402) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.CBASTestExecutor.syncDoExecuteTestFile(CBASTestExecutor.java:237) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.CBASTestExecutor.doExecuteTestFile(CBASTestExecutor.java:214) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.CBASTestExecutor.executeTestFile(CBASTestExecutor.java:196) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.5.0-0000]
at org.apache.asterix.test.common.TestExecutor.executeTest(TestExecutor.java:1705) ~[asterix-app-0.9.5-SNAPSHOT-tests.jar:6.5.0-0000]
at org.apache.asterix.test.common.TestExecutor.executeTest(TestExecutor.java:1676) ~[asterix-app-0.9.5-SNAPSHOT-tests.jar:6.5.0-0000]
at com.couchbase.analytics.test.cbserver.ClusterExecutionITBase.runTest(ClusterExecutionITBase.java:106) ~[test-classes/:?]
at com.couchbase.analytics.test.cbserver.ClusterRebalanceIT.test(ClusterRebalanceIT.java:29) ~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_192-ea]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_192-ea]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_192-ea]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_192-ea]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.12.jar:4.12]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) ~[junit-4.12.jar:4.12]
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) ~[junit-4.12.jar:4.12]
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12]
at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.12.jar:4.12]
at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0]
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0]